Getting to Know the Mel Spectrogram
By: Dalya Gartzman
Tags: Python, Programming, Music, Audio, Data Science

Read this short post if you want to be like Neo and know all about the Mel Spectrogram! (Ho maybe not all, but at least a little)

For the tl;dr and full code, go here.

A Real Conversation That Happened in My Head a Few Days Ago

Me: Hi Mel Spectrogram, may I call you Mel?

Mel: Sure.

Me: Thanks. So Mel, when we first met, you were quite the enigma to me.

Mel: Really? How’s that?

Me: You are composed of two concepts that their whole purpose is to make abstract notions accessible to humans - the Mel Scale and Spectrogram - yet you yourself were quite difficult for me, a human, to understand.

Mel: Is there a point to this one-sided speech?

Me: And do you know what bothered me even more? I heard through the grapevine that you are quite the buzzz in DSP (Digital Signal Processing), yet I found very little intuitive information about you online.

Mel: Should I feel bad for you?

Me: So anyway, I didn’t want to let you be misunderstood, so I decided to write about you.

Mel: Gee. That’s actually kinda nice. Hope more people will get me now.

Me: With pleasure my friend. I think we can talk about what are your core elements, and then show some nice tricks using the librosa package on python.

Mel: Oooh that’s great! I love librosa! It can generate me with one line of code!

Me: Wonderful! And let’s use this beautiful whale song as our toy example throughout this post! What do you think?

Mel: You know you’re talking to yourself right?

The Spectrogram

Visualizing sound is kind of a trippy concept. There are some mesmerizing ways to do that, and also more mathematical ones, which we will explore in this post.

Photo credit: Chelsea Davis. See more of this beautiful artwork here.

When we talk about sound, we generally talk about a sequence of vibrations in varying pressure strengths, so to visualize sound kinda means to visualize airwaves.

But this is just a two dimensional representation of this complex and rich whale song!

Another mathematical representation of sound is the Fourier Transform. Without going into too many details (watch this educational video for a comprehensible explanation), Fourier Transform is a function that gets a signal in the time domain as input, and outputs its decomposition into frequencies.

Let’s take for example one short time window and see what we get from applying the Fourier Transform.

Now let’s take the complete whale song, separate it to time windows, and apply the Fourier Transform on each time window.

Wow can’t see much here can we? It’s because most sounds humans hear are concentrated in very small frequency and amplitude ranges.

Let’s make another small adjustment - transform both the y-axis (frequency) to log scale, and the “color” axis (amplitude) to Decibels, which is kinda the log scale of amplitudes.

Now this is what we call a Spectrogram!

The Mel Scale

Let’s forget for a moment about all these lovely visualization and talk math. The Mel Scale, mathematically speaking, is the result of some non-linear transformation of the frequency scale. This Mel Scale is constructed such that sounds of equal distance from each other on the Mel Scale, also “sound” to humans as they are equal in distance from one another.

In contrast to Hz scale, where the difference between 500 and 1000 Hz is obvious, whereas the difference between 7500 and 8000 Hz is barely noticeable.

Luckily, someone computed this non-linear transformation for us, and all we need to do to apply it is use the appropriate command from librosa.

Yup. That’s it.

But what does this give us?

It partitions the Hz scale into bins, and transforms each bin into a corresponding bin in the Mel Scale, using overlapping triangular filters.