Analyzing Climate Patterns with Self-Organizing Maps (SOMs) | by Haihan Lan | Towards Data Science

In this article, we’ll look at:
The Self-Organizing Map (SOM), and how it can be used in dimensionality reduction and unsupervised learning
Interpreting the visualizations of a trained SOM for exploratory data analysis
Applications of SOMs to clustering climate patterns in the province of British Columbia, Canada
Links to my other articles:
Tensorflow and Custom Loss functionsRandom forestsSoftmax classification
Tensorflow and Custom Loss functions
Random forests
Softmax classification
Back when I was first getting started in learning about neural nets, I came across this curious invention called the Kohonen map, or more commonly called the Self-Organizing Map or SOM in literature. This neural network, inspired by the sensory activation patterns of the human cerebral cortex, trained unsupervised using a simple heuristic method, and was capable of discovering hidden non-linear structure in high dimensional data. Recently I came across a few interesting articles by some climatologists analyzing and predicting weather and climate (a notoriously difficult task) using SOMs, and I thought I’d share with you this interesting unsupervised technique.
Basically, an SOM is a discrete, planar grid of neurons — the hidden layer, sometimes called the Kohnen layer — fed by an input layer:
Each hidden layer neuron has several ‘neighbor’ neurons (the maximum immediate number of neighbors of a single cell can be six as in the hexagonal grid above, or four for a square grid). We define the ‘distance’ between each neuron and its neighbors as the Euclidean distance between the input-to-hidden layer weights of each neuron. The input-to-hidden layer weight of each hidden neuron is also sometimes called a codebook vector. The distance that we have defined will become important later, as it encodes low-dimensional information about the original dataset. In the diagram above, each hidden layer neuron would have a weight vector with n components (i.e. the input is densely connected to the hidden layer).
So how do we train this network? The training procedure is actually quite simple and involves no gradient calculations:
Initialize hidden neuron weights to small random values or use PCA weight initializationFeed data row xi to input layerIterate through each neuron in hidden layer and find the Best Matching Unit (BMU), i.e. the neuron that has the smallest Euclidean distance or metric with the data row xiApply a weight update to the BMU and it’s neighboring neurons. The BMU’s neighbor neurons are calculated using a neighborhood function ΦShrink the neighborhood function ΦRepeat steps 2 to 5 until iteration limit reached or convergence (the average codebook vector distance between all neurons and all data is smaller than some threshold)
Initialize hidden neuron weights to small random values or use PCA weight initialization
Feed data row xi to input layer
Iterate through each neuron in hidden layer and find the Best Matching Unit (BMU), i.e. the neuron that has the smallest Euclidean distance or metric with the data row xi
Apply a weight update to the BMU and it’s neighboring neurons. The BMU’s neighbor neurons are calculated using a neighborhood function Φ
Shrink the neighborhood function Φ
Repeat steps 2 to 5 until iteration limit reached or convergence (the average codebook vector distance between all neurons and all data is smaller than some threshold)
The above algorithm is called competitive learning, and is quite similar in spirit to other heuristics based algorithms such as genetic algorithms and particle swarm methods. A full description of the algorithm is available here. The basic idea of the SOM is to ‘fill the space’ of the underlying data as illustrated by the figure below:
The competitive learning algorithm is typically fast compared to backpropagation on a feedforward networks of similar number of neurons, but can slow down if your SOM grid is very large (104 or more neurons).
SOMs belong to a class of techniques called Non-Linear Dimensionality Reduction (NLDR). Dimesionality reduction techniques (PCA, t-SNE, etc.) are typically used for visualization and interpretation, as well as variable selection. SOMs are better suited for the former application, fitted SOMs can be further analyzed to obtain low-dimensional visualizable and interpretable results to present to your boss, scrum master, marketing team, or your significant other. Furthermore, clustering techniques such as K-means can be applied to the fitted SOM grid, and from that robust clusterings of the input can be obtained. Typically K-means on top of SOM performs a lot better than regular K-means and can sometimes discover patterns of non-linear interactions that are hidden. Let’s now play around with some climate data and see what the SOM can do.
The original SOMPY repo used for the code examples below is available here. SOMPY is a good starting package for Python, but in case you’re also fluent in R check out the kohonen package, it has some very nice visualization tools (also see here for the tutorial on using the kohonen package for R).
The repo containing all the modifications of the SOMPY code for the tasks below with the climate data is here. The main.py script file that runs the code is at https://github.com/hhl60492/SOMPY_robust_clustering/tree/master/sompy/examples.
Historic climate data (monthly summaries) from Jan. 2009 to Dec. 2017 of British Columbia (BC), Canada was collected from http://climate.weather.gc.ca/prods_servs/cdn_climate_summary_e.html. Each CSV file contains the monthly summary of a climate data (such as median, min/max monthly temperature, monthly precipitation amount) and the Latitude and Longitude of the weather station that recorded the data . Each unique weather station gives a single row, and there are 300 or so weather stations in each CSV.
We’ll keep it simpe for now and use the columns Latitude, Longitude, Median Temperature, Max Temperature, Min Temperature and Precipitation. The columns were extracted and coerced to float types. Rows with any NaNs were dropped. SOMPY automatically calculates the size of the SOM grid by calculating the eigenvalues of the data matrix.
The training on 20621 rows of data took about 5 seconds. The first visualization to make note of are the component plane heatmaps.
Each heatmap represents the intensity of a single feature or data column as learned by the SOM grid. Note the heatmap is discretized and each ‘block’ in each subfigure is a unique neuron, the neurons have the same position across all subfigures, i.e. the block on the very top left of the latitude subfigure is the same neuron in the longitude subfigure and the monthly median temperature subgfigure and so on. In general similar heatmaps in different component planes represent correlation of features, dissimilarity represents negligible correlation and inversion represents anti-correlation. Also note that the fitted SOM will represent the features across all months of the year for the entire range of years, i.e. the 9 year average, as sequential data from the months of Jan. to Dec. across 9 years was supplied. Using the previous facts and the latitude and longitude information, we can see that:
The monthly maximum temperature occurs across a variety of latitudes and longitudes, including lat. 49° N and long. 120 °–124° W; this is a large swath of Southern BC close to the US-Canada border, including the southern tip of Vancouver Island, Metro Vancouver, Fraser Valley, and stretching all the way to the Okanagan
The lat. 49° N and long. 120°–124° W area also gets the most precipitation.
The size of the regions in the component plane heatmaps of latitude and longitude show that there is a heavy concentration of weather stations in the region of 49°-50° N and 123°–124° W, where there is the highest concentration of population (The following plot of the latitudes and longitudes of all the weather stations in each CSV file seems to confirm this observation)
(Note: to verify the geography it’s a good idea to actually extract the lat. and long. data from the fitted SOM in Python and label the cells/neurons in the other component plane heatmaps with them.)
Monthly median temperature is correlated with monthly max temperature, and to a lesser extent monthly median temperature.
Precipitation seems anti-correlated with monthly median and max temperature (as a local, I can confirm we have way less rain when it is hot and vice versa–the climate of the Pacific Northwest is such that there is tremendous precipitation during the Late Autumn, Winter and Spring seasons when the temperature is low, and little to no precipitation during the Summer when temperature is high).
The next visualization we can do is called the U-matrix. The U-matrix is also a heatmap but can be interpreted like a topographic map (a map that shows the elevation contours across a region) across our SOM grid. The ‘hills’ in the U-matrix represent large distances between neighbors and vice versa for valleys.
The U-matrix gives us a sense of the landscape of the SOM grid, and what the neighborhoods of the neurons are. It helps us visually see what potential clusterings there are in the SOM grid itself, which is a good sanity check for the result of the K-means clustering which we are going to run on grid. Since K-means is a bit naive on the optimal number of clusters, we’ll use the ‘elbow’ method combined with the Sum of Squared Errors (SSE) to find the optimal number of clusters k. The algorithm for SSE computation is as follows:
For k = 2 to k_end:compute K-means on dataset XFor each cluster n:Compute the L2-norm of the vector difference between each element in cluster n and cluster n’s centroid, and add this to the total SSE
For k = 2 to k_end:
compute K-means on dataset X
For each cluster n:
Compute the L2-norm of the vector difference between each element in cluster n and cluster n’s centroid, and add this to the total SSE
The resulting plot of SSEs vs. different k’s looks something like the following
The elbow point can be found analytically by finding the index of the maximum value of the first difference of the above plot, and adding 1. Unfortunately, the K-means algorithm is not too robust and gives different elbow curves for different runs, so bootstrapping and repeating the elbow/SSE plot analysis over many bootstrap samples will give a better estimate for the optimal k. All the code to do this is in the following gist, the cluster() method in the SOMFactory class (in sompy.py) is modified to implement the above.
For this particular SOM, the maximum value of the first difference of the SSEs averaged across 20 bootstrap samples was 17 and so the optimal k should be 18. Repeating the runs gives use consistent results for the optimal k now, so that’s great. Running the final K-means with this k gives the following clustering:
This resembles the U-matrix plot somewhat, so we are not too far off in terms of the true clustering of the SOM grid.
There’s still a couple of things left to do before we can put on our jackets and go home. Let’s see the features of each cluster centroid (in K-means the centroid is representative of the average features in a cluster), predict which weather station belongs in which cluster, and put the clustered weather stations on a map. We use gmplot and the Google Maps API to do map overlays. By the way, gmplot is woefully outdated and requires some patching to make it work with the current Google Maps API, you also need a valid Google Maps API Key in order to access the API. The gist for the gmplot code modifications is here.
Centroid 0[  51.38661738 -125.27254198    7.51493849   16.8132888    -1.17574106   67.95416645]Centroid 1[  50.61259827 -126.25917862    6.32184038   14.17993615   -1.63282112  340.64034668]Centroid 2[  50.42396495 -125.2544494     7.2744194    16.09718498   -1.19140013  222.65626168]Centroid 3[  50.42534148 -126.39001822    6.72726471   14.53652612   -1.31210139  507.65982791]Centroid 4[  49.89467361 -122.37195052   16.63158605   29.73554202    5.95873865   28.06162618]Centroid 5[  52.27917741 -121.73550782   -3.83946568    8.19646576  -19.93363911   45.30064963]Centroid 6[  50.2945556  -125.21241436    9.15306962   18.32174182    0.93638661  119.43331396]Centroid 7[  50.70282538 -125.32884059    8.00965749   17.614526     -0.77992491  151.77106456]Centroid 8[  50.31580334 -126.23307623    6.5871719    14.64225657   -1.44068436  660.70782628]Centroid 9[  50.28874072 -125.06866063    7.49467144   16.55379438   -0.84254229  251.73801799]Centroid 10[  50.4262205  -126.2629756     6.58541814   14.36302394   -1.36671652  399.90931304]Centroid 11[  50.35982435 -125.51021147    6.32659523   14.73932271   -1.89873252  294.050496  ]Centroid 12[  51.24356927 -121.64231971    6.18712464   18.67948989   -5.48058692   36.06758652]Centroid 13[  50.75675998 -125.60116426    7.48408728   16.37144962   -0.95604598  190.18377378]Centroid 14[  51.1337961  -124.85444455    7.9966279    17.1295218    -0.56629512   94.65084155]Centroid 15[  50.12150383 -126.04955706    6.7603695    14.43437023   -1.13385479  461.59884165]Centroid 16[  50.72982522 -126.73419985    6.88866959   14.84315741   -1.12018377  586.45036101]Centroid 17[  51.44904147 -122.37356247   12.74453517   26.10130055    1.58123601   50.9674971 ]
The components of each centroid correspond to Latitude, Longitude, Median Temperature, Max Temperature, Min Temperature and total Precipitation, across all months of the year.
You can play around with the interactive centroids map here: https://github.com/hhl60492/SOMPY_robust_clustering/blob/master/sompy/examples/centroids_map.html (download to your machine and open with a browser).
One can further analyze the SOM and overlay for example the cluster that each weather station belongs to on the map (one way would be to average the data for each weather stations across all years 2009–2017 and , or perhaps re-fit the SOM for yearly or monthly data only to see the respective yearly and monthly patterns. As well, unfortunately there is a a heavy distribution bias in the location of the weather stations, so some other features to consider would for example be atmospheric pressure measurements, radar data or physical elevation data which have less geographic bias.
To get an idea of how the climatologists are using SOMs in their models, have a look at: http://onlinelibrary.wiley.com/doi/10.1029/2009JD011706/full
We have seen how SOMs can be used for data visualization, discovering correlations and patterns in the data, and performing clustering of the data. To extract more complex patterns, you can feed more input features to the SOM, or partition the features across some dimension such as time. Another thing to note: running K-means clustering should be done with bootstrapped ensembles and the cluster labels matched with something like the Hungarian algorithm, so that stable, consistent clusters are found. Anyways, have fun playing around the SOMs and doing the clusterings.