K-means clustering using Spotify song features | by Alejandra Vlerick | Towards Data Science

The spotify API allows us to create a simple server-side application that accesses user related data from the Spotify app. It also gives you access to information that is not available on the app, such as artist popularity, song metrics, album cover images, etc. It allows you to create, delete and modify existing playlists in a user’s account.
The goal of this project is to use a clustering algorithm to break down a large playlist into smaller ones. For this, song metrics such as ‘danceability’, ‘valence’, ‘tempo’, ‘liveness’, ‘speechiness’ are used.
Data collection using the Spotify API
Finding the ideal number of clusters
Elbow method and silhouette analysis
Feature selection
Clustering algorithm
Playlist creation and performance analysis
If you’re not already a member, get your Medium Membership here!
import spotipyfrom spotipy.oauth2 import SpotifyOAuthfrom spotipy.oauth2 import SpotifyClientCredentials
Connecting to the spotify API was pretty straightforward and the content is pretty well documented. This link will give you all the information you need for connecting and this will give you python sample code.
Go to https://developer.spotify.com/dashboard/ and click Create a Client ID or Create an App to get your “Client ID” and “Client Secret”. After that, Redirect URI must be changed to any page you decide on in the settings of your Spotify application.
scope = "user-library-read playlist-modify-private"OAuth = SpotifyOAuth(        scope=scope,                 redirect_uri='http://localhost:8888/callback',        client_id = 'client_id',        client_secret = 'client_secret',        username= 'username')
The variable “sp” will work as the access key to Spotify data and will allow us to use all the functions to GET and POST requests to and from spotify.
sp = spotipy.Spotify(auth_manager=OAuth)
The next step is to collect the data from my “Liked Songs” Playlist.
The API can only extract 50 songs at a time therefore we run a while loop that goes through the entire playlist in sets of 50. The function used to access my personal liked songs is current_user_saved_tracks().
offset = 0songs = []items = []ids = []while True:    content = sp.current_user_saved_tracks(limit=50, offset=offset)    songs += content['items']    if content['next'] is not None:        offset += 100    else:        break
The following step is to extract the audio features of each song by using the audio_features() function:
for i in songs:   ids.append(i['track']['id'])while index < len(ids):    audio_features += sp.audio_features(ids[index:index + 50])    index += 50features_list = []for features in audio_features:    features_list.append([features['energy'], features['liveness'],                          features['tempo'],features['speechiness'],                          ...                          features['mode'], features['type'],                          features['uri']])df = pd.DataFrame(features_list, columns=['energy', 'liveness',                                          'tempo', 'speechiness',                                          ...                                           'mode', 'type','uri'])
The result is a pandas dataframe that gathers the songs on the rows and their respective features on the columns. Each song is identified by its unique uri.
Just for fun, I decided to look at the distribution of the feature values across all the songs in my playlist:
I concluded very quickly that there is no strong correlation between these metrics. Songs did not follow a set pattern depending on their genre. When trying to perform clustering based of all 12 features, the results were not great... For this I decided to use a lower number of metrics and tested all the possible combinations of features that would yield the best clusters.
Before trying a clustering algorithm on all 12 features, I decided to handpick a few features for the clustering in the hope that it would yield a better cluster quality.
The first step was to normalise all the values such that they lie within [0 ; 1].
from sklearn import preprocessingx = df.values min_max_scaler = preprocessing.MinMaxScaler()x_scaled = min_max_scaler.fit_transform(x)df = pd.DataFrame(x_scaled)
The following step was to find a cluster quality metric based on a selection of a few features. The first thing I did was create a list with all the song metrics and then used the permutations() function to create a set of all possible combinations using three song features at the time.
columns = ["energy",  "speechiness", "acousticness", "instrumentalness", "loudness","tempo","danceability",'valence' , "liveness", "time_signature", "key"]perm = permutations(columns, 3)output = set(map(lambda x: tuple(sorted(x)),perm))
For each permutation I computed a cluster score using the elbow method.
The elbow method is a heuristic approach used in determining the number of clusters in a data set. The method consists of plotting the explained variation as a function of the number of clusters, and picking the elbow of the curve as the number of clusters to use.
For each permutation I extracted an elbow score and an elbow value. The score tells me how good the clustering was for this permutation, the elbow value tells me the ideal number of clusters for that permutation.
model = KMeans(random_state=0)visualizer = KElbowVisualizer(model, k=(2,12), metric='silhouette', timings=False)visualizer.fit(x_scaled)score = visualizer.elbow_score_value = visualizer.elbow_value_
The result is a pandas dataframe of feature permutations and their corresponding ideal number of clusters and the quality score of these clusters.
Once the scores are calculated for each permutation, I decided to pick the ones above >0.4 and to save them in a separate dataframe.
if score>0.4:    idx = df.columns    mylist = idx.tolist()    dict = {        "features": mylist,        "score": score,        "elbow": value    }    df2 = df2.append(dict, ignore_index=True)
There were many permutations that yielded great results with scores even above >0.5.
The permutation [‘instrumentalness’, ‘speechiness’, ‘valence’] suggested creating 4 clusters and had a score of 0.504. This is the one I selected for my playlist creation.
I scaled a dataframe only containing those three metrics and then performed the clustering using the KMeans() function.
from sklearn.cluster import KMeanskmeans = KMeans(init="k-means++",                n_clusters=4,                random_state=15,                max_iter = 500).fit(x_scaled)df1['kmeans'] = kmeans.labels_df1.columns = ['energy', 'instrumentalness', 'loudness','kmeans' ]
The result is a dataframe that contains a song per row with a label [0, 1, 2, 3] corresponding to each cluster that particular song was assigned to. The feature distribution clearly shows that energy and loudness are distributed differently across all clusters.
The 3D scatter plot gives a different perspective showing that higher values of instrumentalness were sorted into the same cluster while energy and loudness differentiated the remaining three clusters.
For more articles like this, find me on Medium here!
If you have any questions, suggestions or ideas on how to improve, please leave a comment below or get in touch through LinkedIn here.
All the code can be found on Github here. The permutations script can be found in analysis_v2.py and the clustering can be found in feature_analysis.py.