Build Your First Mood-Based Music Recommendation System in Python | by Max Hilsdorf | Towards Data Science

While music genre plays an enormous role in building and displaying social identity, the emotional expression of a song and — even more importantly — its emotional impression on the listener is often underestimated in the domain of music preferences.
Only a few decades back, choosing music by genre and/or artist was effectively the only option. Want to hear live music? Well, choose an artist or a — typically genre-based — festival. Want to buy music? Alright, check out our CD shelves categorized by genre and pick an album by an artist you like. This has changed dramatically with the availability of custom or pre-curated playlists and personal recommendations on digital music platforms.
Hence, if your goal is to build a music recommendation system, you will need to find a way to think beyond genre labels and — what is even harder — find quantitative methods to implement your ideas into a real and useful recommendation tool. There are two main branches of recommendation methods:
Here, we model user behavior to predict what a specific user in a specific situation is statistically likely to enjoy. This includes “other people also bought X” as well as taking overall popularity into account. In the end, a random user is statistically likely to enjoy Drake more than Slipknot. Especially if we already know that he likes other Rap artists.
Another approach is to retrieve data from the musical object itself. This can be metadata like release date, an assigned genre, or the label the song was published with. On the other hand, we can also retrieve more or less abstract musical features from the audio itself. This included obvious aspects like key or tempo but also highly abstract mathematical features like MFCCs, which are — in a way — a measurement of musical timbre (GER: “Klangfarbe”, EN: “color of sound”).
In this post, I am going to show you an easy way to build an effective recommendation system without any machine learning. Well... at least none from your side.
Extracting audio data using existing algorithms or machine learning models from MIR (music information retrieval) research is a difficult thing to do as a beginner in the domain. Luckily, some smart people built APIs to share their data with us, so that we can build innovative technology with it.
The Spotify Web API is a great place to start. It is free to use and as you can see in Figure 1, it offers multiple mood features extracted right from their internal machine learning models. But how in the world are we going to model the complexity of moods with just “danceability”, “valence”, “energy”, and “tempo” (the latter hardly qualifying as a mood feature)? Actually, we only need two of those.
One of the dominant psychological models of the factor structure of emotions (i.e. how many independent dimensions of emotions exist) is the Valence-Arousal Plane (Figure 2). It is a two-dimensional model reducing every existing emotion or mood to its arousal component (is the feeling high- or low energy?) and its valence component (does it feel good or bad?). Although this model is not without its problems (e.g. fear and anger would be located right next to each other here), it is widely adopted because it strikes a great balance between complexity and predictive power.
The amazing thing here is that we can nicely adopt the Valence-Arousal Plane model using the “valence” and “energy” features from the Spotify API. In the rest of the article, I will go over how to crawl the data from the API and how to implement a simple and effective recommendation system with it.
Let us start with the least fun part: To access the Spotify API, you need to register an application following this guide. Alternatively, check out this medium post where the process is nicely illustrated. I know you just want to get started, but you will have to work through the authentification first, unfortunately. Once you have your Client ID and your Client Secret ID, I will provide you with all the code you need to create a recommendation dataset. Does that sound like a deal?
To build a recommendation system, we need a database of tracks out of which we can pick the best fits to a given track. For this, you are only going to need the Python package tekore (<pip install tekore> in your terminal), your Spotify Client ID and Secret ID, and the code I will provide you with.
First, please create a new python script called “authorization.py” in your project directory and fill it with this code:
import tekore as tkdef authorize(): CLIENT_ID = “ENTER YOUR CLIENT ID HERE” CLIENT_SECRET = “ENTER YOUR CLIENT SECRET HERE” app_token = tk.request_client_token(CLIENT_ID, CLIENT_SECRET) return tk.Spotify(app_token)
Just enter your Client ID and Client Secret ID where it says so in the code. This script only does one thing: It authorizes your access to the Spotify API and returns an object that allows you to access the API. The script serves as a helper script for our main program. Mainly, however, it ensures that your Client ID and Secret ID are hidden from readers/users of the main program. Only this allows me to share my entire code with you.
There are, of course, plenty of ways to collect a dataset. We are going to use a method that draws 100 tracks from over 120 genres, resulting in a track database of around 12000 tracks with music from various styles.
You are going to need to install the packages pandas and tqdm through <pip install [name of package]> if you do not have them installed already. From there, just copy and run the code below and/or follow my quick overview of the code.
The algorithm works like this: We authorize our API access using the helper script. Then, we get all of Spotify’s 120 genre labels using <sp.recommendation_genre_seeds()> and set the number of recommendations per genre to 100 (which is the maximum amount possible). In the next step, we set up a dictionary that can hold all the data we want to crawl. Next, we loop through each of the genres and then through each track in the genre. For each of these tracks, we crawl metadata and audio information and store them in the <data_dict> we created earlier. Then, we transform the dictionary to a pandas dataframe, drop duplicate id’s, and finally export the dataframe to our working directory.
AMAZING! You know have your dataset with the name “valence_arousal_dataset.csv” right in your working directory. Next, we are going to use this dataset to build our recommendation system.
Our similarity measurement is easy to understand if you look at Figure 3 for a minute. There, I have estimated where a couple of well-known songs would be located on the Valence-Arousal Plane. In this plane, each vector (a track with its coordinates “valence” and “arousal”), in this example Toto’s “Rosanna”, can be connected with each other track by a line. Visually, it becomes clear that the emotional profile of Michael Jackson’s “Thriller” is more similar to “Rosanna” than all the other tracks which are even further away in the plane.
Mathematically, the lines connecting each track are also vectors and we can measure their length. This length or “norm” of a 2D vector (a,b) is defined as sqrt(a2+b2). Let us say a track has 0.5 “valence” and 1 “energy” and thus the coordinates (0.5, 1). Then, its length, that is its distance from the origin of the coordinate system (point (0,0)) is equal to sqrt((0.5)2 + 12) which is roughly 1.12. Thus, all we need is to find all the vectors (arrows) connecting a given track with all the other tracks, apply the formula and take the arrow with the lowest norm/length.
This “arrow vector” from point p1 to point p2 is given by p2-p1. Hence, the mood-based “distance” between a track t1 and a track t2 is equal to norm(t2-t1). In other words, you subtract t1 from t2 and then compute the norm of the resulting vector using the formula above. In Python code, this is simple to implement:
def distance(p1, p2):    distance_x = p2[0]-p1[0]    distance_y = p2[1]-p1[1]    distance_vec = [distance_x, distance_y]    norm = (distance_vec[0]**2 + distance_vec[1]**2)**(1/2)    return norm
In my code, I use numpy.linalg.norm(p2-p1), which does the same thing.
Before we jump into the actual implementation of the recommendation system, I would like to point you to two statistical problems our system is going to have. If you do not really care, just skip this part or come back later.
As you can see in Figure 4, the two features are completely differently distributed. While “valence” seems to follow something approximating a very flat normal distribution, “energy” is heavily left-skewed. This has one major downside for our model: Without going into too much statistical detail, this means: A “jump” of 0.2 in “valence” is not always the same thing as a jump of 0.2 in “energy”. However, our model assumes that the vector (0.5,0.5) is just as close to (0.7,0.5) as it is to (0.5,0.7). If you want to resolve this conflict, applying a z-transform can help. For the purpose of this post, we will ignore this issue.
Another problem is illustrated in Figure 5. In psychological theory, valence and arousal are supposed to be two statistically independent dimensions of emotion. However, if we plot all of the 12,000 tracks in our dataset into the “energy-valence” plain, we see that as “valence” increases, so does “energy”. In fact, a regression slope of 0.250 is found, which points to a non-neglectable correlation between the two features. For this, there is, unfortunately, no solution as this error (at least for our purposes) is hard-built into the Spotify API.
Building the final recommendation algorithm is simple from this point on. I am going to guide you through the last steps in this section. You can also follow along with this notebook.
To start off, we import some modules:
import pandas as pdimport randomimport authorization # this is the script we created earlierimport numpy as npfrom numpy.linalg import norm
Next, we read in our dataframe with the ~12000 tracks:
df = pd.read_csv("valence_arousal_dataset.csv")
Now, we combine the “valence” and “energy” columns to form a single vector “mood_vec” per track:
df["mood_vec"] = df[["valence", "energy"]].values.tolist()
The last step before we implement the recommender algorithm is to authorize your Spotify API access, just like we did earlier:
sp = authorization.authorize()
Finally, it is time to implement our vector-distance-based recommendation algorithm. The steps, this algorithm takes, are:
Crawl the input track’s “valence” and “energy” values from the Spotify APICompute the distances between the input track and ALL other tracks in our reference dataset.Sort the reference tracks from lowest to highest distance.Return the n least distant tracks.
Crawl the input track’s “valence” and “energy” values from the Spotify API
Compute the distances between the input track and ALL other tracks in our reference dataset.
Sort the reference tracks from lowest to highest distance.
Return the n least distant tracks.
The code is:
What is left to do is to test whether our algorithm produces meaningful results! Each Spotify track as an ID, which you can simply extract from the web browser like this:
This will give you “https://open.spotify.com/track/3JOVTQ5h8HGFnDdp4VT3MP?si=96f7844315434b0a” where “3JOVTQ5h8HGFnDdp4VT3MP” is the track ID.
So how about we throw “Mad World” (“valence” = 0.30, “energy”= 0.06) by Gary Jules into our recommender?
mad_world = "3JOVTQ5h8HGFnDdp4VT3MP"recommend(track_id = mad_world, ref_df = df, sp = sp, n_recs = 5)
The top recommendation with “valence” = 0.31 and “energy” = 0.05 is “Glory Manger” by Harry Belafonte.
Not bad! Although “Glory Manger” is from a completely different genre, our algorithm has matched the songs because they both have similar levels of “valence” and “energy”.
Let us try one more example! “Rosanna” by Toto has a “valence” of 0.739 and an “energy” of 0.513.
The top recommendation is “Sentimientos De Chartón” by Duelo with a “valence” of 0.740 and an “energy” of 0.504.
Again, the genre is very different, being more of a “latin” than a “pop-rock” track. Yet, “Sentimientos De Cartón” captures this sense of “romantic longing” while still conveying a sense of movement and groove, just like “Rosanna”.
Congratulations! If you have followed the post up to this point, you have either built your first mood-based recommendation system or hopefully at least learned a bit about the way you could approach programming such a system. There are, however, still a few things worth briefly reflecting on or discussing.
Together, we built the basis of a music recommendation system that uses an adaptation of the “Valence-Arousal Plane” alongside vector-distance measurements to match tracks that convey a similar type of emotion/mood. This approach is fundamentally different from collaborative filtering methods (“other users also bought [...]”) since it attempts to extract the inner qualities of a song without relying on user data. This is especially helpful for those of you who (like me) do not operate a large music platform and have billions of relevant user data points. Another plus is that this method’s implementation is fast and free, from collecting the data to implementing the actual recommendation algorithm. Finally, recommending music across genre categories is a desirable quality of any real-world recommender system, since this supports the user in engaging with new kinds of music and developing their personal music taste.
As mentioned in the statistics section, performing a “z-transform” on the “valence” and “energy” features will make your distance measurements and hence your recommendations more accurate.
The Spotify API offers more features like “Danceability”, “Acousticness”, and “Tempo”. Try to think of reasonable ways to expand on the “Valence-Arousal Plane” or to develop your own set of variables from scratch.
Combining the mood-based approach with a collaborative-filtering approach can also be fun to explore. How about, after we compute the top 10 recommendations using “valence” and “energy”, we reorder the recommendations by “Popularity” (which is also a feature in the Spotify API).
You could let the user decide on a set of genres of which all recommendations have to be or a set of genres that are not supposed to show up in the recommendations.
Lastly, you can expand your reference dataset by finding exploring the possibilities of the Spotify API and analyzing more and more tracks. A larger database makes it more likely that you have a near-perfect match ready for every input track.
Thank you very much for reading this article! Let me know if you have any questions or comments or if anything is not working right. I am currently exploring creative ways to expand the recommendation system shown here using self-built machine learning models. Stay tuned if you would like to see that!