How to Create Large Music Datasets Using Spotipy | by Max Hilsdorf | Towards Data Science

Are you a music lover or a programmer? Chances are, you’re both, just like me! When I started using Spotipy, I had little programming experience and wanted to explore computational audio analysis. Now, as I’m knee-deep into programming and Data Science, I’m starting to appreciate Spotipy for creating amazing datasets for my data science projects.
you are a data scientist or programmer wanting to generate interesting datasets for machine learning
you are a musician/musicologist wanting to learn programming or data analysis
Spotipy is a Python library that makes it easier for users to access the Spotify Web API and retrieve all kinds of music data from it. When I started using Spotipy, it was the first time I ever got in touch with an API. Therefore, if you have never used API’s, don’t worry. You can do it!
Work with your own user data (create or edit your playlists, find your favorite tracks etc.)Get data for every track, album, and artist on Spotify
Work with your own user data (create or edit your playlists, find your favorite tracks etc.)
Get data for every track, album, and artist on Spotify
In this guide, we’ll explore the possibilities of the latter application. I’m going to show you how to use this data to create amazing datasets for statistical analyses or machine learning projects.
As Spotify has over 50 Million songs, the possibilities to create large datasets are endless. On top, you’ll be able to retrieve the data very quickly, once you’ve set up the basics.In the domain of music, there are so many amazing applications of machine learning that you can explore. Has predicing Titanic survivors gotten boring? How about you build a genre classifier, a recommender system, or a hit song predictor? To my ear, that sounds much cooler.
As Spotify has over 50 Million songs, the possibilities to create large datasets are endless. On top, you’ll be able to retrieve the data very quickly, once you’ve set up the basics.
In the domain of music, there are so many amazing applications of machine learning that you can explore. Has predicing Titanic survivors gotten boring? How about you build a genre classifier, a recommender system, or a hit song predictor? To my ear, that sounds much cooler.
Caught your interest? Great! After reading this story, you are able to create your first music dataset with a huge sample size (1k, 10k, 100k? You decide!). To make this easy for you, I’ll guide you through the 3 steps you need to take to get there.
This is the hard part, but stick with me. We’ll be coding soon! In order to use the Spotify Web API, you’ll need to register an application at https://developer.spotify.com/. It doesn’t matter if you are actually building an application or just exploring the API. These are your 3 steps to success:
Login with your Spotify account or register one for free at https://developer.spotify.com/dashboard/loginGo to your Dashboard and click “Create a Client ID”
Login with your Spotify account or register one for free at https://developer.spotify.com/dashboard/login
Go to your Dashboard and click “Create a Client ID”
Here, you come up with an application name (it doesn’t matter what you call it) and a description for it. When I created mine, I was just being honest and wrote that I just wanted to explore the API with my app. It worked for me.
3. Open your app view and click “Edit Settings”. Enter http://localhost/ in the Redirect URLs field.
4. Go back to your app view and click “Show Client Secret” just under your Client ID.
5. Store the Client ID and the Client Secret ID in a text file or something to quickly access them later. We’ll need them for the authorization in Spotipy.
In case my way of explaining wasn’t helpful to you, I recommend you check out different resources for this part. A great introduction is this article by Max Tingle.
In case the last part was a bit confusing or annoying for you, be sure that you’ll be through the boring part in just a handful lines of code.
First, you’ll need to make two imports. Install spotipy using the command “pip install spotipy” first, if you haven’t already.
First, you’ll need to make two imports. Install spotipy using the command “pip install spotipy” first, if you haven’t already.
import spotipyimport spotipy.util as util
2. Next, assign your Client and Secret ID to variables.
CLIENT_ID = "YourClientID"CLIENT_SECRET = "YourSecretID"
3. I won’t go through the next lines of code in detail. I suggest you copy-paste them and learn more later.
token = util.oauth2.SpotifyClientCredentials(client_id=CLIENT_ID, client_secret=CLIENT_SECRET)cache_token = token.get_access_token()sp = spotipy.Spotify(cache_token)
Great! Now, you’ll get started with analyzing some actual music in step 2.
Now that the Spotify API lies at your feet, what are you gonna do with it? As suggested earlier, analyzing a big number of tracks makes for a great dataset. However, hand selecting every single track would be painful. You’ll need some collection or list of tracks. Two options that proved useful in my personal experience are searching for tracks by playlist or artist. Both have their pros and cons which I will go over quickly before we get into the real analysis.
How big do playlists get? This beauty above has 10,000 songs, which seems like the maximum playlist size Spotify allows for. Maybe you want to use the “Biggest Playlist Ever” playlist with 5,000 songs , too. Look around and you’ll find many of these large playlists which you can use to build a large dataset quickly. However, you’ve got to ask yourself what your actual research interest is. Will this collection of 15,000 be of any use to anyone? Maybe you want to use such a dataset to investigate how some audio features are distributed on Spotify?
Another way would be to choose playlists based on a theme. You could take the first 20 search results for playlists with the word “sleep” in their title. Maybe you want to compare the songs in “sleep” playlists to songs in “concentration” playlists to find out in which ways they differ. I have used this approach before to analyze whether romance and heartbreak songs differ musically (which they don’t, surprisingly).
If you have selected the playlists that are interesting to you, make sure to store their creators username as well as the playlist id. You can retrieve the ID from the url as seen below.
Another way to select music would be through their artist. I’ve used this approach before to build a machine learning model that assigned the music from the newly released Tool album “Fear Inoculum” to the correct band out of 6 progressive metal/rock bands. In that case, you’re dealing with low sample sizes, however. There are few artists with multiple hundreds of songs available, not to mention thousands. Spotify has a “recommended artists” feature, however. You might want to use it to increase your sample sizes while still using similar music.
Once you’ve chosen a couple of artists, store their artist ID in the same way you did with the playlist ID before.
For step 3, I’m going to show you how you could go about analyzing music from playlists. You are encouraged to try out how to do this via artist yourself. Feel free to ask me any questions, as well.
Let’s get into some actual coding. The basic Spotipy function to retrieve information for every track in a playlist is:
sp.user_playlist_tracks("username", "playlist_id")
Let’s use the “Warum Fuzzy Feeling” Playlist by Spotify for this example.
Taking the ID from the url, we can now apply the function.
sp.user_playlist_tracks("spotify", "37i9dQZF1DX5IDTimEWoTd")
The output is overwhelming and, at first glance, totally uncomprehensible for anyone not used to dealing with APIs. This is some of the output.
While I highly encourage you to explore the output yourself, I won’t go in depth on this here and simply show you which features you want and where you can find them. If you do want to explore the output yourself, it’s useful to treat it like a collection of multiple dictionaries nested inside each other.
These are the kinds of features we can extract from the output.
Metainformation (Artist, Album, Track Name, Track ID)Acoustic Parameters (Loudness, Key, Mode, Tempo)Psychoacoustic Parameters (Danceability, Energy, Instrumentalness, Liveness, Valence)
Metainformation (Artist, Album, Track Name, Track ID)
Acoustic Parameters (Loudness, Key, Mode, Tempo)
Psychoacoustic Parameters (Danceability, Energy, Instrumentalness, Liveness, Valence)
Spotify has some great resources for finding out more about the features we can extract (see figure below). Learn more here.
Because this is an introductory article, I am going to show you the extraction function with only brief explanations of how exactly it works.
Create an empty dataframe with all relevant columnsStore sp.user_playlist_tracks(“username”, “playlist_id”) to a “playlist” variableLoop through every track in the playlistCreate an empty dict to fill with track informationExtract metadata directlyExtract audio features through sp.audio_features(track_id)Concatenate the track_dict onto the existing dataframeReturn the final dataframe
Create an empty dataframe with all relevant columns
Store sp.user_playlist_tracks(“username”, “playlist_id”) to a “playlist” variable
Loop through every track in the playlist
Create an empty dict to fill with track information
Extract metadata directly
Extract audio features through sp.audio_features(track_id)
Concatenate the track_dict onto the existing dataframe
Return the final dataframe
Here is the function as pure code. You can find the entire code from this tutorial in a more appealing format in this github repository.
def analyze_playlist(creator, playlist_id):        # Create empty dataframe    playlist_features_list = ["artist","album","track_name",  "track_id","danceability","energy","key","loudness","mode", "speechiness","instrumentalness","liveness","valence","tempo", "duration_ms","time_signature"]        playlist_df = pd.DataFrame(columns = playlist_features_list)        # Loop through every track in the playlist, extract features and append the features to the playlist df        playlist = sp.user_playlist_tracks(creator, playlist_id)["items"]    for track in playlist:        # Create empty dict        playlist_features = {}        # Get metadata        playlist_features["artist"] = track["track"]["album"]["artists"][0]["name"]        playlist_features["album"] = track["track"]["album"]["name"]        playlist_features["track_name"] = track["track"]["name"]        playlist_features["track_id"] = track["track"]["id"]                # Get audio features        audio_features = sp.audio_features(playlist_features["track_id"])[0]        for feature in playlist_features_list[4:]:            playlist_features[feature] = audio_features[feature]                # Concat the dfs        track_df = pd.DataFrame(playlist_features, index = [0])        playlist_df = pd.concat([playlist_df, track_df], ignore_index = True)            return playlist_df
At this point, you can copy my function, edit it, or explore the API youself. In any case, the function above returns a dataframe, which can easily be converted into any data format you want.
# csvdf.to_csv("dataframe.csv", index = False)# exceldf.to_excel("dataframe.xlsx", index = False)
Lastly, I’m going to show you how you can create a dataframe containing multiple playlists.
You can use the following function for this task. It analyzes every single playlist, adds a “playlist” column with the playlist name. Lastly, the function concatenates the old and new dataframe.
def analyze_playlist_dict(playlist_dict):        # Loop through every playlist in the dict and analyze it    for i, (key, val) in enumerate(playlist_dict.items()):        playlist_df = analyze_playlist(*val)        # Add a playlist column so that we can see which playlist a track belongs too        playlist_df["playlist"] = key        # Create or concat df        if i == 0:            playlist_dict_df = playlist_df        else:            playlist_dict_df = pd.concat([playlist_dict_df, playlist_df], ignore_index = True)                return playlist_dict_df
However, you’ll need to organize your playlists in a certain format in order for the function to work. As you can see below, you’ll need to store the playlists in a dictionary with the playlist names as keys and their creator and playlist id as values in the form of a tuple.
playlist_dict = { “warm_fuzzy_feeling” : (“spotify”, “37i9dQZF1DX5IDTimEWoTd”),  “love_songs_heart” : (“indiemono”, “5KbTzqKBqxQRD8OBtJTZrS”), “romance_songs” : (“Susan Doles”, “7sAUK3XK8NHH1s5vGcTBkF”)}
Let’s see what happens if we run the function.
multiple_playlist_df = analyze_playlist_dict(playlist_dict)multiple_playlist_df["playlist"].value_counts()
Output:
love_songs_heart      100romance_songs          77warm_fuzzy_feeling     70Name: playlist, dtype: int64
As you can see, using this method, you can easily analyze 20 large playlists and get multiple thousands of track analyses. Keep in mind, however, that you’ll need to do some data cleaning. Especially, checking for duplicates will be an important step to take.
You’ve got your first music dataset now, but what to do next? Here are some ideas:
Analyze playlists for different genres, moods, or activities. Store each groups playlist (f.e. “Sad” vs. “Happy”) in a separate dataframe. Then, use logistic regression, decision trees, or deep learning approaches like neural networks or random forests to build a classification algorithm.
Analyze music based on different artists instead of based on playlists. You’ll find the following two functions useful for this:
sp.audio_analysis(track_id)sp.audio_features(track_id)
Build a recommender system that recommends a user music based on a genre, mood, or activity input.
Thank you for reading this article! If anything is missing or doesn’t work for you, feel free to message me.