Content-based Automated Music Playlist Generation using Deep Learning
By: G Pravin Shankar
Tags: Music Recommendations, Artificial Intelligence, Machine Learning, Data Science, Deep Learning

Photo by Andy Kelly on Unsplash

Objective

The detailed step-by-step implementation of a content-based music recommender system is explained in this article. The entire code is accessible in my GitHub. Basic knowledge about recommender systems and deep learning is assumed.

Table

Making Sense of Sound Digital Audio Auditory Perception Music Why is Music Entertaining? Overview Problem Statement Featurization Existing Solution Data First-cut approach Final Features Sample Recommendations Feature Analysis Failed Experiments Future Work References

Making Sense of Sound

Sound is our perception of the vibration of surrounding air molecules(underwater audio perception is beyond the scope of this article). The medium where it originates could be solid, liquid, or gas.

Photo by Gentrit Sylejmani on Unsplash

What we hear when someone swims nearby is nothing but the vibration of water molecules which in turn vibrates its surrounding air molecules.

If we hear vibrating air molecules, one might wonder, why don’t we hear when we swing our hands in the air. It is because our auditory system perceives vibrations with frequency only in the range of 20 Hz to 20 kHz.

When we drop a stone, we hear only when it hits a solid or a liquid surface. Even though the stone in motion disturbs the air molecules, the frequency is well below our hearing range. However, when it hits the water, the neighboring air molecules vibrate at the same frequency at which the water molecules vibrate, making it audible. The origin of this sound(or vibration) is the stone striking the water and it travels through the air to reach our ears.

Sound propagates through air molecules from left to right[Source: Dr. Daniel A. Russell]

The gray line represents the vibrating source, dots both red and black are all air particles(molecules) and to understand the actual movement of these particles look at any one of the red dots. They don’t move like luggage placed on the carousel, rather its a back and forth motion. Only the vibration travels from the gray line to the rightmost end.

When a particle moves from point ‘A’ to point ‘B’ and comes back to ‘A’, it is said to complete one cycle. Total no of cycles per second is nothing but the frequency in hertz scale and amplitude is the distance between point ‘A’ and point ‘B’.

The frequency or frequencies at which an object tends to vibrate with when hit, struck, plucked, strummed, or somehow disturbed is known as the natural frequency of the object. If the amplitudes of the vibrations are large enough and if natural frequency is within the human frequency range, then the vibrating object will produce sound waves that are audible. [Source: The physics classroom]

Most living things can sense certain vibrations of objects which indicates the significance of sound for survival. We communicate and emote efficiently by causing these vibrations which propagate away from its source in all directions. Generally, these vibrations are too fast to be seen, and unlike vision, the audition span is not restricted to certain degrees. This explains the widespread use of sirens to indicate emergencies even though the light is extremely fast.

Digital Audio

The microphone contains a thin diaphragm similar to our eardrum which is a transducer that converts mechanical energy to electrical energy.

In very simple terms, it vibrates at the same frequency of the vibration of the surrounding air particles, and the amplitude of this vibration is converted to electrical energy. This continuous(analog) information is converted to digital form by sampling 44100 values per second at equal time intervals and representing each value in 16 bits. The digital format facilitates storage, access, and manipulation of data.

Loudspeaker reproduces the sound by performing precisely the opposite operation of the microphone.

Auditory Perception

Human interpretation of amplitude and frequency are loudness and pitch, but the relationship is not linear. It is quite analogous to how the difference between one million dollars and one million one hundred dollars doesn’t matter when you buy a house. We can easily differentiate 50 Hz and 100 Hz frequencies, but not 10,000 Hz and 10,050 Hz.

The Mel scale approximates our perception of frequency which is roughly linear till 500 Hz and logarithmic afterward; Similarly, sound pressure(amplitude) is expressed in decibel, a logarithmic unit, to approximate our perception, and 0 dB is the threshold for human hearing.

Apart from pressure, loudness, measured in phon, is also dependant on our auditory system’s ability to amplify certain frequencies.

At 0 dB, only frequencies between 2 kHz to 5 kHz can be heard whereas to sense a 30 Hz frequency it needs to be more than 50 dB. Decibel and phon are equal for the frequency of 1 kHz.

Phon vs decibel [Source: Wikipedia]

Music

Music, a fine art, is nothing but the artificial vibrations of air particles that are aesthetically pleasing to humans but has no practical purpose; It, however, has the potential for medical use. Every vibration processed by the auditory cortex of the brain triggers both physical and psychological responses.

Music establishes mood which influences people’s perception (especially visual perception[Jolij J, Meurs M (2011)]), attitude, and behavior. It has always been an integral part of rituals all around the world and even during the silent film era, music played a major role.

Video explains how music can completely alter our perception

From a composer’s point of view, music is a sequence of harmonies. The first step in music composition is to choose the key and scale, which influences the song’s genre and emotion; This determines the notes available to compose the melody. A music note represents frequency(or, more precisely, pitch) and its duration.

A melody(or tune) is composed by arranging a sequence of notes but only one at a time. It is the most notable, critical, and repetitive part of a song.

The final step is to harmonize the melody. To every note in a melody, notes are added, for example, of a lower frequency to enhance the melody. When multiple notes played together, provide a satisfying effect they are said to be in harmony.

Melody is represented by orange notes in the video

One important thing to note is that the single-frequency vibrations are extremely rare in nature. Even musical instruments produce vibration with frequencies in addition to the intended note and these less dominant additional frequencies along with the envelope distinguishes the musical instruments.

For example, ‘middle C’ has a frequency of 261.6, sounds different when played in different instruments.

Video shows the vibration caused by different instruments when the same note is played

Why is Music Entertaining?

According to Yerkes-Dodson law, the performance of a task is dependant on the arousal of an organism. Performance improves with the increase in arousal, but beyond a saturation point, it starts reducing or stops increasing depending on the complexity of the task.

Hebbian version of the Yerkes–Dodson law[Source: Diamond DM, et al. (2007)]

This “Inverted u” relationship is also seen in our music preferences, discovered by Daniel Ellis Berlyne.[source: Chmiel, A., & Schubert, E. (2017)]

The sequential nature of music creates an opportunity to predict the next event. It is because of our ability to recognize and memorize patterns. When the prediction is closer we feel bored as there is only a little information to learn(too little arousal) and when the prediction is poor we feel annoyed as there is overwhelming information to learn(too much arousal). The maximal reward or pleasure is experienced somewhere in-between.

Melody often repeats in a song and also in our mind after we listen, this helps the process of memorization and explains why our favorite song gets boring or an uninteresting song becomes enjoyable after we hear it many times. Simply, our exposure to music determines our preference in music.

Learn the rules like a pro, so you can break them like an artist. [Pablo picasso]

Following the music theory and subtly breaking it is what helps the musicians to achieve optimal arousal in their audience.

To gain a better understanding, read this article by Robert Zatorre.

Overview

Major music streaming service providers have rights to millions of songs that are organized and displayed alluringly to simplify the user experience. It takes more than one’s lifetime to hear them all and the probability of a user hearing a particular song is two-sided, the user’s day-to-day experiences and the algorithm(or hybrid of algorithms) cashed in by the company.

The art of personalization is paramount in this data-intensive era of the internet to engage and guide the audience in their quest.

Metadata used by the content-based filtering don’t normally expand beyond the high-level abstraction but handles new users. Collaborative filtering, the other widely used and more efficient approach needs users’ historical data and periodic retraining to overcome its fundamental limitation of assuming that similar users evolve similarly.

Also, people are growing increasingly uncomfortable with the risk of personal data being exploited in the process and would prefer an alternative. Users at least in the premium tier should have the convenience to use the service without providing access to their data.

This is an attempt to analyze the scope of deep learning methods on audio where no other data will be used to mimic a bit of the human auditory system with limited resources and no compromises on users’ privacy.

Problem Statement

Generate playlist based on the given query track(s) without using any metadata like genre, sub-genre, artist, year of release, etc.

Featurization

As mentioned before, a digital audio file contains air pressures(amplitudes) sampled in equal time intervals, and our auditory system process this information for reasoning.

107852.mp3

If you ask me what is the difference between classical music and rock music, I would say classical music uses piano and violin whereas rock music uses electric guitar and acoustic drums. So, to differentiate two genres, I broke the songs into simpler sounds(eg: piano) and then compared those sounds to find similarity between songs in the same genre and also differences between two genres.

Fourier transform(article and video explanations) can be used to convert a sound into multiple basic sounds or more technically complex vibration into multiple single frequency vibrations. But, to represent a sound sequence, we need to do Fourier transform for every event and obtain the amplitude of all the frequencies(in continuous scale) which is not practical.

We can break the song into multiple overlapping windows and assume each window as an event and apply the Fourier transform, and also bin the frequencies. For every window, this will give us the amplitude of each frequency bin which can be arranged in time to build a spectrogram.

Spectrogram with frequency in the Mel scale and amplitude in decibel approximates the information received by the auditory cortex of the brain. Today, it is the most widely used representation of audio in deep learning.

The vertical axis represents 216 ‘Mel frequency’ bins, the horizontal axis represents ‘time’ in seconds and the color represents ‘amplitude’ in decibels(black implies no or very low amplitude and yellow implies very high amplitude).

Mel spectrogram of “107852.mp3”

Existing Solution

Recommending music on Spotify with deep learning by Sander Dieleman:

Situations like cold start and lack of data breed alternative lines of thoughts for an otherwise effective collaborative filtering. The Convolutional neural networks, state-of-the-art image processing models were utilized to address these limitations by using collaborative filtering based representations to supervise the learning process. Latent representations are normalized to minimize the impact of the popularity of the songs.

Each song split into multiple frames was represented as a spectrogram with amplitude in the logarithmic scale and frequency in the Mel scale stacked vertically to represent time. The rectangular kernels were convolved only in time.

The sequential nature of audio was partially discarded and instead overall presence of features was extracted.

One of the CNN architectures used by Sander Dieleman

The trained model could compute feature representations of all the new tracks added every day(speculated to be roughly 40,000 at Spotify) directly and any appropriate similarity measure could be used for recommendations. However, the latent features used in the supervision need to be recomputed to adapt to changing user preferences and this necessitates the frequent retraining of the entire neural network.

Data

Roughly, 66 hours of audio(8,000 tracks of length 30 seconds each) forms the data. Metadata and commonly used features are also available. [Source: FMA].

First-cut Approach

Mel frequency cepstrum coefficients(MFCCs) were the state-of-the-art representation for audio before the emergence of deep learning. Here MFCC vector represents each window of a song instead of a 216-dimensional vector. Initially, there were, say, 2048 samples in a window and were represented using 216 values in the Mel spectrogram. They are further reduced to 20 here.

MFCCs are the representation of sound converted into multiple intermediate sounds instead of basic sounds. The intermediate sounds are made up of groups of basic sounds separated by equal intervals, similar to how notes in harmony are separated. Musical note(s) played on one or more instruments may be an intermediate sound.

MFCCs are obtained by applying discrete cosine transform on the decibel scale dependant values in the Mel frequency bins of a window.

Each song is windowed 1293 times, a total of 20*1293 values or a low-resolution grayscale spectrogram(cepstrogram) to represent them. The rows are standardized to avoid the domination of certain coefficients and to improve discrimination of time windows in a spectrogram and also different spectrograms.

MFCC color spectrogram of “107852.mp3”

Standardized MFCC color spectrogram of “107852.mp3'

Autoencoder was used to extract an n-dimensional vector representation from the spectrogram. Convolution based model only had a slightly better mean squared error(MSE) than a baseline model, but the representation made sense in differentiating songs whereas encodings of the baseline model which predicts zeros would have been useless.

The Kernels were convolved in both time and MFCC axis on a low-resolution grayscale spectrogram(matrix of size 20*1293*1) containing vectors of amplitudes instead of a color spectrogram, which additionally saved memory and computations.

One of the autoencoder models

Sample input(X1)

Sample output(Y1^)

Recommendation based on the vector representation obtained from the autoencoder felt random and also didn’t work for genre prediction.

Final Features

Mel spectrogram, as commonly believed, turned out to be the best representation. VGG16 pre-trained on the Imagenet dataset was used to extract the feature representation of the tracks. The final layer is replaced with a Max pooling layer and then the output is flattened to get a 25,600-dimensional sparse vector.