A Brief Introduction to Self-Organizing Maps
By: Thought Partner For Data
Tags: Data Visualization, Som, Clustering, Self Organizing Map, Dimensionality Reduction

A Brief Introduction to Self-Organizing Maps

Photo by Andrew Stutesman on Unsplash

Self-Organizing Map (SOM) is one of the common unsupervised neural network models. SOM has been widely used for clustering, dimension reduction, and feature detection. SOM was first introduced by Professor Kohonen. For this reason, SOM also called Kohonen Map. It has many real-world applications including machine state monitoring, fault identification, satellite remote sensing process, and robot control [1]. Moreover, SOM is used for data visualization by projecting higher dimensional data into lower dimensional space leveraging topological similarity properties. In the process of model development, SOM utilizes competitive learning techniques while the conventional neural network applies an error reduction process through backpropagation with a gradient descent algorithm. In this post, I share my understanding of SOM, how it learns, methods, and limitations of the SOM.

How does SOM learn?

The architecture for SOM is very similar to the Neural network architecture but it is much simpler than the artificial neural network (ANN). SOM one contains two layers: an input layer and an output layer (feature map). SOM starts feature mapping by initializing a weight vector. Weights in SOM have a completely different connotation than in ANN. In modeling a neural network: activation function is applied to the linear combination of the weights and input values to produce output for each of the neurons in the architecture. On the other hand, SOM does not apply the activation function. and employ weights as the characteristics of the neuron in the architecture. The weights are generally randomly generated. The motivation of using the weight vector as neuron characteristics is to push each of the row (observations) of the given data into an imaginary space where each of the rows acts as a point. This is the core of the SOM. Once, each of the data holds an imaginary point into the input space, then a search for closest points (data).

Assume that we are given with a 3-dimensional dataset (features — x1, x2, and x3) that has only four observations (rows). We want to apply the SOM method on the dataset for dimension reduction purposes, i.e. we want to transform the data into a 2-dimensional space from given 3-dimensions. There are several parameters in the SOM such as the number of neurons. We choose two neurons as we want to transform the data into 2-dimensional space. That means, the SOM contains two layers: an input layer (3 neurons/nodes) and an output layer (2 neurons/nodes). Here, we present several steps of SOM:

1. initialize random weight vector — in this step a random weight vector is initialized. A good initial approximation is a well-known challenge for iterative methods like ANN. Though random initialization of weights is a common practice, the principal component initialization, which leverages the first principal component space of the data, has gained immense popularity due to the reproducibility of the results [3]. Neurons’ weight vector dimension/size should be the size of the input space. In this example: the neurons W1 and W2 are associated with the weight vector (1, 2, 3) and (6, 5, 4) respectively.

2. randomly select an input — assume the randomly selected input is the last row (1,1,1)

process of computing winner neuron

3. compute winner neuron using Euclidean distance — in this step, the Euclidean distance between the randomly selected input and each of the neurons is calculated. The following table shows the calculation of the two neurons. The neuron with the shortest distance is the winner neuron. Here, W1 is the winner neuron. The winning neuron is called the Best Matching Unit (BMU). Instead of Euclidean distance, we can use other distance metrics to calculate BMI as well. In this step, the neighborhood of the BMU calculated where the neighborhood size is considered based on an exponential decay function [2]. The size of the neighborhood around the BMU decreases over time. The following equation shows the exponential decay function where t is the current time step, sigma0 is the radius at the initial time, is the time constant that depends on, and a selected number of iterations for the method.

exponential decay function for finding the neighbor size

shrinking of the radius of the neighborhood of the BMU

4. update neuron weights — in this step both the BMU and neighborhoods’ weights are adjusted to investigate the most similar neuron to the input vector and expect a significant change for neighbors closer to the BMU. The neighbors learn less (insignificant change in the weights) that are further away from the BMU. The following equation shows the calculation for update neurons:

the equation for updating neurons weights

Where t is the current time step and L is the learning rate which is also an exponential decay function like the previous one:

the equation of learning rate

The n is the influence rate (another exponential decay function) that decides the radius (size) of the neighborhood and with the iteration increases the size gradually shrinks and eventually the neighborhood shrunk to size zero at the end of the iteration (training)[2]. The following equation is the expression of influence rate:

the equation for influence rate

Here, the distance is the lattice distance between each pair of the neurons (W_i and W_j).

5. repeat 2–4 until training finish — Iterate steps 1–4 until the positions of the neurons are stabilized.

Though SOM is a widely used easily interpretable unsupervised technique that allows visualizing the similarity between observations within the low-dimensional grid, it has some limitations:

SOM requires a sufficient amount of data for generating meaningful clusters as insufficient or extraneous data might add additional randomness to the clusters. SOM encounters difficulties clustering with categorical data [5]. The SOM generally applied to the categorical variable after transforming the data into an indicator variable (like a binary variable). Similarity properties among the categorical features may lose valuable information due to the transformation and consequently, a trained SOM might not be able to present the proper topological properties of the data.

Reference:

[1] https://users.ics.aalto.fi/jhollmen/dippa/node28.html#applications

[2] https://www.saedsayad.com/clustering_som.htm

[3] A. Ciampi, Y. Lechevallier, Clustering large, multi-level data sets: An approach based on Kohonen self-organizing maps, in D.A. Zighed, J. Komorowski, J. Zytkow (Eds.), PKDD 2000, Springer LNCS (LNAI), vol. 1910, pp. 353–358, 2000.

[4]https://www.cs.hmc.edu/~kpang/nn/som.html#:~:text=The%20main%20drawback%20of%20the,be%20entirely%20accurate%20or%20informative.

[5] https://www.comp.nus.edu.sg/~rudys/arnie/tnn-som-categoricaldata.pdf