K-Means Clustering Using Spotify Data
By: Rafael Duarte
Tags: K Means, Clustering, Songs, Data Science, Spotify

Analyzing Spotify data and clustering songs with K-Means

According to their own website, Spotify is a digital music, podcast, and video streaming service that gives you access to millions of songs and other content from artists all over the world.

AI and Data Science

Like most modern technology companies, Data Science plays a big role. Since its goal is to connect artists and their audience, recommendation systems play a big role, and you can experience that every time you use the app.

The Discover Weekly, Daily Mixes, and our yearly Spotify Wrapped are some of the examples we can name.

Have you ever searched for a song and ended up finding lots of similar songs you loved and saved instantly? That’s also their algorithm at work.

The Project

Being an avid Spotify user myself, there’s one feature I think is lacking. I like to call it Mood Playlist, which would be some kind of Biased Random mode.

Whenever you’re listening to your saved songs and you want to shake things up a little, it’s a great idea to turn on Random mode. It really is random, and sometimes that’s really what you want.

However, sometimes you just want to set the mood for the songs to be played, using just your saved songs. That’s exactly what this feature would do. You select a song, hit the mode, and Spotify queues up the most related songs, in order to keep up with your mood.

Imagine you’re on a dinner date and want to listen to those romantic songs you have saved. Of course, you haven’t prepared a playlist beforehand. All you have to do is select “The way you look tonight”, switch Mood Playlist on, and it will play all of those songs that are similar to the first one you selected.

The goal of this project is to explore the data, understand the correlations, and come up with an algorithm that would serve as the basis for the creation of the Mood Playlist feature.

To do that, we’ll be using a dataset obtained by me using the Spotify API. A notebook on the process to get the data from Spotify using the Python Library Spotipy can be found here.

Also, kudos to tomigelo, for coming up with the code I used to retrieve the data from Spotify.

If you’re interested in the full project, with all the code and a more in-depth approach, click here.

Machine Learning

For the situation we’re facing, we need to categorize our data points into clusters, and then use these to gather these data points, which in our case are songs, into a sequence of songs, that will become our Mood Playlist.

Using as reference this post on the Towards Data Science blog:

“Clustering is a Machine Learning technique that involves the grouping of data points. Given a set of data points, we can use a clustering algorithm to classify each data point into a specific group. In theory, data points that are in the same group should have similar properties and/or features, while data points in different groups should have highly dissimilar properties and/or features. Clustering is a method of unsupervised learning and is a common technique for statistical data analysis used in many fields.”

Among the top Clustering techniques, Towards Data Science mentions:

K-Means Clustering

Mean-Shift Clustering

Density-Based Spatial Clustering of Applications with Noise (DBSCAN)

Expectation–Maximization (EM) Clustering using Gaussian Mixture Models (GMM)

Agglomerative Hierarchical Clustering

For this project, we’ll be using K-Means.

Preparing Our Data For Clustering

For the sake of this post, I won’t be going through our exploratory data analysis. If you’re interested in all of the many features made available by Spotify through their API, and how they correlate, influence popularity and many other things, make sure to check out the full project with code.

There, I analyzed the most famous songs in Brazil, including an analysis of what features contribute more to popularity, top songs and artists, and a lot more. It’s really worth your while, but not the focus of this article. So let’s get back to it.

Elbow Method — Finding Out The Right Number Of Clusters

For K-Means we have to set a number of clusters for our data to be divided in. Since this is, at first, a completely arbitrary task, we have to look for options to make it as precise as possible.

One of the most popular methods to do so is the Elbow Method. Basically, we calculate different scenarios for different numbers of clusters and then plot them in a line.

The number we’re looking for is the position where the line starts to flatten, making it look like an elbow for the plot. After some testing, I’ve figured out that 5 clusters yield better results.

Data Preprocessing

To make the model perform better, it’s important to preprocess our data. To better explain the reason behind this step, I’ll quote Edupristine:

“Distance computation in k-Means weights each dimension equally and hence care must be taken to ensure that unit of dimension shouldn’t distort relative near-ness of observations.”

After testing, I found that MinMaxScaler yielded the best results.

K-Means

Now that our data is ready, we can start working with K-Means to cluster our data points.

The goal here is to organize the songs in clusters, where similar songs are put together.

Clusterized Songs

Since we’re working with many features, our visualization is not as simple as the one from the first picture.

Either way, we can still understand the visualization and have an idea of what the clusters look like.

Mood Playlist

The kmeans.predict() method yields an array with the assigned cluster for each row in our dataset. To make these predictions easier to grasp, let’s transform them into a data frame and concatenate it to our original dataset as a new column.