Deep Clustering for Financial Market Segmentation | by Yuefeng Zhang, PhD | Towards Data Science

Unsupervised learning, supervised learning and reinforcement learning are three main categories of machine learning methods. Unsupervised learning has many applications such as clustering, dimensionality reduction, etc. The machine learning algorithms K-means and Principal Component Analysis (PCA) are widely used for clustering and dimensionality reduction respectively. Similarly to PCA, the T-distributed Stochastic Neighbor Embedding (t-SNE) is another unsupervised machine learning algorithm for dimensionality reduction. t-SNE is typically used for embedding high-dimensional data in a two or three dimensional space for data visualization.
With the advancement of unsupervised deep learning, the Autoencoder neural network is now frequently used for high dimensionality (e.g., a dataset with thousands or more features) reduction. Autoencoder can also be combined with supervised learning (e.g., Random Forest) to form Semi-supervised learning method (see deep patient as an example).
Recently a Deep Embedded Clustering (DEC) method [1] was published. It combines autoencoder with K-means and other machine learning techniques for clustering rather than dimensionality reduction. The original implementation of DEC is based on Caffe. An implementation of DEC in Keras for MNIST dataset can be found in [2].
In this article, similarly to [2], I implement the DEC algorithm in Keras and use the public dataset Kaggle Credit Card Dataset for Clustering [3] to show how to use the new implemented DEC model to cluster the credit card dataset for customer segmentation. The rest of this article is arranged as follows:
Data Preparation
Implementation of the DEC Method in Keras
Summary
This section describes the common data preprocessing steps required for clustering.
After the Kaggle credit card dataset [3] has been downloaded onto a local machine, it can be loaded into Pandas DataFrame as follows:
import Pandas as pddata = pd.read_csv('./data/CC_GENRAL.csv')data.head()
It can be seen from the DataFrame above that CUST_ID field is unique for each customer data record. This field with unique values is not useful for clustering and thus can be dropped:
data_x = data.drop(['CUST_ID'], axis=1)
It can also be seen from the DataFrame that the ranges of values are very different for different fields/features. It is well known that K-means is sensitive to the scale of feature values because it uses Euclidean distance as similarity metrics. To avoid this issue, the values of all features are rescaled into the range of [0, 1]:
from sklearn.preprocessing import MinMaxScalernumeric_columns = data_x.columns.values.tolist()scaler = MinMaxScaler() data_x[numeric_columns] = scaler.fit_transform(data_x[numeric_columns])data_x.head()
The following code is to check if any missing data exists in the dataset:
data_x.isnull().sum()
The above table shows that there are one missing CREDIT_LIMIT record and 313 missing MINIMUM_PAYMENTS. In this case, it makes sense to fill up missing data with zero:
data_x.fillna(0, inplace=True)
Similarly to [2], the DEC algorithm in [1] is implemented in Keras in this article as follows:
Step 1: Estimating the number of clusters
Step 2: Creating and training a K-means model
Step 3: Creating and training an autoencoder
Step 4: Implementing DEC Soft Labeling
Step 5: Creating a new DEC model
Step 6: Training the New DEC Model
Step 7: Using the Trained DEC Model for Predicting Clustering Classes
Step 8: Jointly Refining DEC Model
Step 9: Using Refined DEC Model for Predicting Clustering Classes
Step 10: Comparing with K-means
As described before, the DEC method combines Autoencoder with K-means and other machine learning techniques. In order to train a K-means model, an estimated number of clusters is required. The number of clusters is estimated in this article by exploring the silhouette values of different K-means model executions:
for num_clusters in range(2,10):    clusterer = KMeans(n_clusters=num_clusters, n_jobs=4)    preds = clusterer.fit_predict(x)    # centers = clusterer.cluster_centers_    score = silhouette_score (x, preds, metric='euclidean')    print ("For n_clusters = {}, Kmeans silhouette score is {})".format(num_clusters, score))
A silhouette value measures how similar a data record is to its own cluster (cohesion) compared to other clusters. The silhouette value ranges from −1 to +1, where a high value indicates that the data record matches to its own cluster well and matches poorly to its neighboring clusters.
The silhouette values above indicate that the top two choices of the number of clusters are 2 and 3. The number of clusters of 3 is chosen in this article.
Once the number of clusters is determined, a K-means model can be created:
n_clusters = 3kmeans = KMeans(n_clusters=n_clusters, n_jobs=4)y_pred_kmeans = kmeans.fit_predict(x)
In addition to K-means, an autoencoder is required as well in the DEC algorithm [1]. The following function is to create an autoencoder:
def autoencoder(dims, act='relu', init='glorot_uniform'):    n_stacks = len(dims) - 1        input_data = Input(shape=(dims[0],), name='input')    x = input_data        # internal layers of encoder    for i in range(n_stacks-1):        x = Dense(dims[i + 1], activation=act,  kernel_initializer=init, name='encoder_%d' % i)(x)    # latent hidden layer    encoded = Dense(dims[-1], kernel_initializer=init, name='encoder_%d' % (n_stacks - 1))(x)    x = encoded    # internal layers of decoder    for i in range(n_stacks-1, 0, -1):        x = Dense(dims[i], activation=act, kernel_initializer=init, name='decoder_%d' % i)(x)    # decoder output    x = Dense(dims[0], kernel_initializer=init, name='decoder_0')(x)        decoded = x        autoencoder_model = Model(inputs=input_data, outputs=decoded, name='autoencoder')    encoder_model     = Model(inputs=input_data, outputs=encoded, name='encoder')        return autoencoder_model, encoder_model
An autoencoder model is created as follows:
n_epochs   = 100batch_size = 128dims = [x.shape[-1], 500, 500, 2000, 10] init = VarianceScaling(scale=1. / 3., mode='fan_in',                           distribution='uniform')pretrain_optimizer = SGD(lr=1, momentum=0.9)pretrain_epochs = n_epochsbatch_size = batch_sizesave_dir = './results'autoencoder, encoder = autoencoder(dims, init=init)
As described in [1], the sizes of layers [500, 500, 2000, 10] are chosen as a generic configuration of the autoencoder neural network for any dataset.
The diagram of the resulting encoder model can be created as below:
from keras.utils import plot_modelplot_model(encoder, to_file='encoder.png', show_shapes=True)from IPython.display import ImageImage(filename='encoder.png')
The autoencoder is trained as follows:
autoencoder.compile(optimizer=pretrain_optimizer, loss='mse')autoencoder.fit(x, x, batch_size=batch_size, epochs=pretrain_epochs)autoencoder.save_weights(save_dir + '/ae_weights.h5')
The weights of the trained autoencoder are saved for later usage:
autoencoder.save_weights(save_dir + '/ae_weights.h5')autoencoder.load_weights(save_dir + '/ae_weights.h5')
One of the key components in the DEC method [1] is soft labeling, that is, assigning an estimated class to each of the data samples in such a way that it can be refined iteratively. To this end, similarly to [2], a new ClusteringLayer class is defined:
class ClusteringLayer(Layer):    def __init__(self, n_clusters, weights=None, alpha=1.0, **kwargs):        if 'input_shape' not in kwargs and 'input_dim' in kwargs:            kwargs['input_shape'] = (kwargs.pop('input_dim'),)        super(ClusteringLayer, self).__init__(**kwargs)        self.n_clusters = n_clusters        self.alpha = alpha        self.initial_weights = weights        self.input_spec = InputSpec(ndim=2)    def build(self, input_shape):        assert len(input_shape) == 2        input_dim = input_shape[1]        self.input_spec = InputSpec(dtype=K.floatx(), shape=(None, input_dim))        self.clusters = self.add_weight(name='clusters', shape=(self.n_clusters, input_dim), initializer='glorot_uniform')                 if self.initial_weights is not None:            self.set_weights(self.initial_weights)            del self.initial_weights        self.built = True    def call(self, inputs, **kwargs):        q = 1.0 / (1.0 + (K.sum(K.square(K.expand_dims(inputs, axis=1) - self.clusters), axis=2) / self.alpha))        q **= (self.alpha + 1.0) / 2.0        q = K.transpose(K.transpose(q) / K.sum(q, axis=1))                 return q    def compute_output_shape(self, input_shape):        assert input_shape and len(input_shape) == 2        return input_shape[0], self.n_clusters    def get_config(self):        config = {'n_clusters': self.n_clusters}        base_config = super(ClusteringLayer, self).get_config()        return dict(list(base_config.items()) + list(config.items()))
Once the soft labeling layer is defined, it can be used to form a DEC model as follows:
clustering_layer = ClusteringLayer(n_clusters, name='clustering')(encoder.output)model = Model(inputs=encoder.input, outputs=clustering_layer)
A diagram of the new DEC model can be created as below:
from keras.utils import plot_modelplot_model(model, to_file='model.png', show_shapes=True)from IPython.display import ImageImage(filename='model.png')
The new DEC model can be compiled as follows:
model.compile(optimizer=SGD(0.01, 0.9), loss='kld')model.get_layer(name='clustering').set_weights([kmeans.cluster_centers_])
The new DEC model is trained iteratively:
# computing an auxiliary target distributiondef target_distribution(q):    weight = q ** 2 / q.sum(0)    return (weight.T / weight.sum(1)).Tloss = 0index = 0maxiter = 1000 update_interval = 100 tol = 0.001 # tolerance threshold to stop trainingindex_array = np.arange(x.shape[0])for ite in range(int(maxiter)):    if ite % update_interval == 0:        q = model.predict(x, verbose=0)        p = target_distribution(q)  idx = index_array[index * batch_size: min((index+1) * batch_size, x.shape[0])]    loss = model.train_on_batch(x=x[idx], y=p[idx])    index = index + 1 if (index + 1) * batch_size <= x.shape[0] else 0
As described in [1], the training procedure above iteratively refines the clusters by learning from the high confidence assignments with the help of the auxiliary target distribution function target_distribution(). Specifically, the DEC model is trained by matching the soft assignment to the target distribution. To this end, in the DEC model the objective/loss function is defined as a Kullback-Leibler (KL) divergence loss between the soft assignments and the auxiliary distribution.
The model weights of the trained model are saved for later usage:
model.save_weights(save_dir + '/DEC_model_final.h5')model.load_weights(save_dir + '/DEC_model_final.h5')
Once the DEC model is trained, then it can be used for predicting clustering classes as follows:
q = model.predict(x, verbose=0)p = target_distribution(q) y_pred = q.argmax(1)
A silhouette score of 0.291 is obtained as follows:
from sklearn.metrics import silhouette_scorescore = silhouette_score(x, y_pred, metric='euclidean')
The following code can be used to use t-SNE to embed the dataset into a two dimensional space and then use color coding of the predicted clustering labels to visualize the predicted clustering result:
import numpy as npfrom sklearn.manifold import TSNEx_embedded = TSNE(n_components=2).fit_transform(x)vis_x = x_embedded[:, 0]vis_y = x_embedded[:, 1]plt.scatter(vis_x, vis_y, c=y_pred, cmap=plt.cm.get_cmap("jet", 256))plt.colorbar(ticks=range(256))plt.clim(-0.5, 9.5)plt.show()
Figure 1: Clustering of the new DEC model with silhouette score of 0.291.
The main idea behind the DEC method [1] is to simultaneously learn the feature representations and cluster assignments using deep neural networks. To this end, the following code uses the pre-trained autoencoder and K-means model to define a new model that takes the preprocessed credit card dataset as input and output both the predicted clustering classes and decoded input data records.
autoencoder, encoder = autoencoder(dims, init=init)autoencoder.load_weights(save_dir + '/ae_weights.h5')clustering_layer = ClusteringLayer(n_clusters, name='clustering')(encoder.output)model = Model(inputs=encoder.input, outputs=[clustering_layer, autoencoder.output])
The diagram of the joined model can be created as follows:
from keras.utils import plot_modelplot_model(model, to_file='model.png', show_shapes=True)from IPython.display import ImageImage(filename='model.png')
The refinement of the DEC model is performed as follows:
kmeans = KMeans(n_clusters=n_clusters, n_init=20)y_pred = kmeans.fit_predict(encoder.predict(x))model.get_layer(name='clustering').set_weights([kmeans.cluster_centers_])y_pred_last = np.copy(y_pred)model.compile(loss=['kld', 'mse'], loss_weights=[0.1, 1], optimizer=pretrain_optimizer)for ite in range(int(maxiter)):    if ite % update_interval == 0:        q, _  = model.predict(x, verbose=0)        p = target_distribution(q)          y_pred = q.argmax(1)        # check stop criterion        delta_label = np.sum(y_pred != y_pred_last).astype(np.float32) / y_pred.shape[0]        y_pred_last = np.copy(y_pred)        if ite > 0 and delta_label < tol:            print('delta_label ', delta_label, '< tol ', tol)            print('Reached tolerance threshold. Stopping training.')            break    idx = index_array[index * batch_size: min((index+1) * batch_size, x.shape[0])]    loss = model.train_on_batch(x=x[idx], y=[p[idx], x[idx]])    index = index + 1 if (index + 1) * batch_size <= x.shape[0] else 0
The jointly refined model weights are saved:
model.save_weights(save_dir + '/b_DEC_model_final.h5')model.load_weights(save_dir + '/b_DEC_model_final.h5')
The following code is to use the refined DEC model for predicting clustering classes:
q, _ = model.predict(x, verbose=0)p = target_distribution(q)  y_pred = q.argmax(1)
The code below can be used to reuse the t-SNE embedded two dimensional space (vis_x, vis_y) and use the color coding of the new predicted clustering labels to visualize the new predicted clustering result:
plt.scatter(vis_x, vis_y, c=y_pred, cmap=plt.cm.get_cmap("jet", 256))plt.colorbar(ticks=range(256))plt.clim(-0.5, 9.5)plt.show()
Figure 2: Clustering of the refined DEC model with silhouette score of 0.318.
Figure 3 shows the result of clustering by K-means. By comparing Figure 3 with Figure 2, we can see that K-means achieved relatively higher silhouette score. However, it can be seen that the refined DEC model predicted clusters with more clearly separable boundaries.
Figure 3: Clustering of the K-means model with silhouette score of 0.372.
In this article, similarly to [2], I implemented a new DEC model in Keras based on the original DEC algorithm in [1] and then applied the new model to the public dataset Kaggle Credit Card Dataset for Clustering [3].
The model evaluation results demonstrated that the new refined DEC model predicted more clearly separable clusters of the credit card dataset compared with the K-means method. This new DEC model has the potential of being used for credit card customer segmentation, other financial market segmentation, etc.
A Jupyter notebook with all of the source code used in this article is available in Github [4].
[1] J. Xie, R. Girshick, A. Farhadi, Unsupervised Deep Embedding for Clustering Analysis, May 24, 2016
[2] Chengwei, How to do Unsupervised Clustering with Keras
[3] Kaggle Credit Card Dataset for Clustering
[4] Y. Zhang, Jupyter notebook in Github
DISCLOSURE STATEMENT: © 2019 Capital One. Opinions are those of the individual author. Unless noted otherwise in this post, Capital One is not affiliated with, nor endorsed by, any of the companies mentioned. All trademarks and other intellectual property used or displayed are property of their respective owners.