Spectral graph clustering and optimal number of clusters estimation
By: Madalina Ciortan
Tags: Graph, Clustering, Machine Learning

Spectral graph clustering and optimal number of clusters estimation

This post explains the functioning of the spectral graph clustering algorithm, then it looks at a variant named self tuned graph clustering. This adaptation has the advantage of providing an estimation for the optimal number of clusters and also for the similarity measure between data points. Next, we will provide an implementation for the eigengap heuristic computing of the optimal number of clusters in a dataset based on the largest distance between consecutive eigen values of the input data’s laplacian.

Now let’s start by introducing some basing graph theory notions.

Adjacency matrix (A)

Given a graph with n vertices and m nodes, the adjacency matrix is a square n*n matrix with the property:

A[i][j] = 1 if there is an edge between node i and node j, 0 otherwise

Because A is symmetric, its eigen vectors are real and orthogonal (the dot product is 0).

Degree matrix (D)

The degree matrix is a n*n diagonal matrix with the property

d[i][i] = the number of adjacent edges in node i or the degree of node i

d[i][j] = 0

Laplacian matrix (L)

The laplacian matrix is a n*n matrix defined as: L = D -A

Its eigen values are positive real numbers and the eigen vectors are real and orthogonal (the dot product of the 2 vectors is 0)

Conductance

A measure of the connectivity of a group to the rest of the network relative to the density of the group (the number of edges that point outside the cluster divided by the sum of the degrees of the nodes in the cluster). The lower the conductance, the better the cluster.

Calculating the eigen values and eigen vectors of A with x ( n dimensional vector with the values of the nodes): A * x = lambda * x

The spectrum of a matrix representing the graph G is a set of eigenvectors xi of the graph ordered by the corresponding eigen values lambda i.

Now that we introduced the most important building blocks of graph theory, we are ready to summarize the spectral clustering steps:

Compute the Laplacian matrix L of the input graph G Compute the eigen values (lambda) and eigen vectors (x) such that

L* x = lambda * x

3. Select n eigenvectors corresponding to the largest eigenvalues and redefine the input space as a n dimensional subspace

4. Find clusters in this subspace using various clustering algorithms, such as k-means

It is also possible to use instead of the adjacency matrix defined above an affinity matrix which determines how close or similar are 2 points in our space. As defined in the sklearn implemenatation:

similarity = np.exp(-beta * distance / distance.std())

A good resource demoing the creation of the affinity matrix is this youtube video.

Both Spectral Clustering and affinity propagation have been implemented in python. This jupiter notebook shows a quick demo of their usage.

clustering = SpectralClustering(n_clusters=nb_clusters, assign_labels="discretize", random_state=0).fit(X)

y_pred = clustering.labels_

plt.title(f'Spectral clustering results ')

plt.scatter(X[:, 0], X[:, 1], s=50, c = y_pred);

Spectral clustering is a technique known to perform well particularly in the case of non-gaussian clusters where the most common clustering algorithms such as K-Means fail to give good results. However, it needs to be given the expected number of clusters and a parameter for the similarity threshold.

Self tuning Spectral Clustering

The idea behind the self tuning spectral clustering is determine the optimal number of clusters and also the similarity metric σi used in the computation of the affinity matrix.

As explained in this paper the affinity matrix is defined as

where d(si, sj ) is some distance function, often just the Euclidean distance between the vectors si and sj. σ is the scale parameter and is a measure of the similarity between points. Usually it is selected manually. It can also be set automatically by running the clustering many times with different values and selecting the one producing the least distorted cluster. This paper suggest to calculate a local scaling parameter σi for each data point si instead of a single scaling parameter. The paper proposes to analyse the neighborhood of each point si and thus define: σi = d(si, sK) where sK is the K’th neighbor of point si. This is illustrated in the figure below, taken from the original paper, for a value of K=7.

The affinity matrix with local scaling can be implemented as follows:

A second way to estimate the number of clusters is to analyze the eigenvalues ( the largest eigenvalue of L will be a repeated eigenvalue of magnitude 1 with multiplicity equal to the number of groups C. This implies one could estimate C by counting the number of eigenvalues equaling 1).

As shown in the paper:

Another type of analysis can be performed on the eigenvectors but it is not in the scope of this post.

Eigengap heuristic for finding the optimal number of clusters

This paper A Tutorial on Spectral Clustering — Ulrike von Luxburg proposes an approach based on perturbation theory and spectral graph theory to calculate the optimal number of clusters. Eigengap heuristic suggests the number of clusters k is usually given by the value of k that maximizes the eigengap (difference between consecutive eigenvalues). The larger this eigengap is, the closer the eigenvectors of the ideal case and hence the better spectral clustering works.

The code base for this post can be found on Github.

Resources