How to Easily Process Audio on Your GPU with TensorFlow | by David Schwertfeger | Towards Data Science

Deep Learning on audio data often requires a heavy preprocessing step.
While some models run on raw audio signals, others expect a time-frequency presentation as input. Preprocessing is often done as a separate step, before model training, with tools like librosa or Essentia.
But, as you start working with larger datasets, this workflow presents a challenge.
Anytime you change parameters, like sample-rate or FFT-size, you need to process the whole dataset again before you can resume training.
And that means waiting. 😴
Even when parallelized onto available CPU-cores, preprocessing can take a long time. Plus, you need to consider how to store and access files for different parameters. This undoubtedly wastes disk space and mental resources and can quickly become a headache.
Does any of this sound familiar?
Tired of this tedious two-step process, I started to ask myself if there wasn’t a better way.
“Isn’t there a better way?”
I recently set up an efficient audio-data pipeline which enables me to load audio from file paths into models on-demand.
towardsdatascience.com
And I wanted to use the same data pipeline for spectrogram-based models, too.
In this post, I want to share with you:
How to leverage the power of your GPU for your signal processing tasks.
How to build a custom preprocessing-layer to use in any neural network.
And a little bonus at the end. 😲
Read on to find out more.
A popular feature representation across audio-domains in Deep Learning applications is the mel-spectrogram.
A mel-spectrogram is a visual representation of a signal’s frequency spectrum over time. The main difference to a standard spectrogram is that frequencies are projected onto the mel-scale, where the perceived distance of pitches equals the distance of mel-frequencies. This is inspired by how we hear.
Likewise, the mel-spectrogram’s magnitudes are generally logarithmically scaled because this is closer to how we perceive changes in loudness. So, a more precise term would be log-magnitude mel-scaled spectrogram. But because this is quite a mouthful, most people call it log-mel-spectrogram or mel-spectrogram for short. It’s worth pointing out, though, that although mel refers to the frequency-scale, log describes the scale of the magnitudes.
So, how can you transform your raw audio signals into mel-spectrograms?
Compute the short-time Fourier transform of your audio signalsCompute the magnitudesInstantiate the mel filterbankWarp the linear-scale magnitude-spectrograms to mel-scaleTransform magnitudes to log-scale
Compute the short-time Fourier transform of your audio signals
Compute the magnitudes
Instantiate the mel filterbank
Warp the linear-scale magnitude-spectrograms to mel-scale
Transform magnitudes to log-scale
Let’s look at each step in detail.
The short-time Fourier transform (STFT) divides a long signal into shorter segments, often called frames, and computes the spectrum for each frame. The frames typically overlap to minimize data-loss at the edges. Joining the spectra for each frame creates the spectrogram.
To compute the STFT with TensorFlow, use tf.signal.stft(signals) where signals is a tensor containing your audio signals.
Some parameters you need to set are:
frame_length: The length of each frame in samples. This is often called window-length or window-size. The window-size trades temporal resolution (short windows) for frequential resolution (long windows).
frame_step: The number of samples between frames. This is often called hop-length or hop-size.
fft_length: The size of the FFT to apply. This is often called FFT-size and matches the frame_length. It defaults to the smallest power of 2 that can enclose a frame. So, if frame_length is a power of 2 and you don't explicitly set the fft_length, it takes on the same value.
spectrograms = tf.signal.stft(signals,                              frame_length=1024,                              frame_step=512)
The STFT from the previous step returns a tensor of complex values. Use tf.abs() to compute the magnitudes.
magnitude_spectrograms = tf.abs(spectrograms)
We can now plot the magnitude-spectrogram.
Note that you won’t see much before properly scaling the magnitudes, though. The second subplot is scaled with librosa.amplitude_to_db(). So, technically, it's a log-magnitude power-spectrogram.
More about that in step 5.
Transforming standard spectrograms to mel-spectrograms involves warping frequencies to the mel-scale and combining FFT bins to mel-frequency bins.
TensorFlow makes this transformation easy.
You can create a mel-filterbank which warps linear-scale spectrograms to the mel-scale with tf.signal.linear_to_mel_weight_matrix().
You only need to set a few parameters:
num_mel_bins: The number of mel-frequency bands in the resulting mel-spectrogram.
num_spectrogram_bins: The number of unique spectrogram-bins in the source spectrogram which equals fft_length // 2 + 1.
sample_rate: The number of samples per second of the input signal.
lower_edge_hertz: The lowest frequency in Hertz to include in the mel-scale.
upper_edge_hertz: The highest frequency in Hertz to include in the mel-scale.
Multiply the squared magnitude-spectrograms with the mel-filterbank and you get mel-scaled power-spectrograms.
mel_power_specgrams = tf.matmul(tf.square(magnitude_spectrograms),                                mel_filterbank)
We perceive changes in loudness logarithmically. So, in this last step, we want to scale the mel-spectrograms’ magnitudes logarithmically, too.
One way to do that would be taking the log of the mel-spectrograms. But that might get you into trouble because log(0) is undefined. Instead, you want to convert the magnitudes to decibel (dB) units in a numerically stable way.
log_magnitude_mel_spectrograms = power_to_db(mel_power_spectrograms)
Here’s how to do that in TensorFlow based on librosa.power_to_db.
After conversion to log-scale, the spectrograms’ maximum values are zero and the smallest values negative top_db.
Now here comes the fun part.
Combining the individual steps into a custom preprocessing layer allows you to feed raw audio to your network and compute mel-spectrograms on-the-fly on your GPU.
To create your mel-spectrogram layer (or any custom layer), you subclass from tf.keras.layers.Layer and implement three methods:
__init__(): Save the layer's configuration in member variables.build(): Define your weights.call(): Perform the logic of applying the layer to input tensors. This is where you transform your audio input-tensors to mel-spectrograms.
__init__(): Save the layer's configuration in member variables.
build(): Define your weights.
call(): Perform the logic of applying the layer to input tensors. This is where you transform your audio input-tensors to mel-spectrograms.
Here’s my implementation of a custom Keras layer that transforms raw audio to log-mel-spectrograms:
Once defined, you can add audio preprocessing to your neural networks in one line (8).
Now, you may be wondering if it isn’t incredibly inefficient to transform the same audio signals into spectrograms again and again.
And that’s a good point.
While writing this post, I came across Kapre, a library of “Keras layers for audio and music signal preprocessing.” The authors, Choi et al., present a benchmark that shows, their preprocessing layers add about 20 percent overhead.
So, it’s a trade-off. Like almost anything in life.
In this case, you trade storage space for slightly longer computation time. But not only that. You also get rid of a tedious two-step process and instantly gain more flexibility in training your Deep Learning models on audio data.
Trying different preprocessing parameters is now as simple as restarting your training routine with different parameters.
No need to maintain a separate preprocessing-script.
No need to remember changing parameters in more than one place.
No need to process the whole dataset before you can continue training.
Is it worth the overhead?
You decide for yourself.
All the code and examples are available in this Colab notebook:
colab.research.google.com
Now it’s your turn.
What is a workflow you have simplified (or like to simplify) that is giving you more flexibility and mental space, although it’s slightly more expensive?
[1] H. Purwins et al., Deep Learning for Audio Signal Processing (2019), IEEE Journal of Selected Topics in Signal Processing 13.2: 206–219
[2] K. Choi et al., Kapre: On-GPU Audio Preprocessing Layers for a Quick Implementation of Deep Neural Network Models with Keras (2017) arXiv preprint