Fixed-Point DSP for Data Scientists | by Sandeep Mistry | Towards Data Science

Authors: Sandeep Mistry, Henri Woodcock on behalf of the Arm Software Developer’s Team
When Machine Learning (ML) is applied in audio systems, the input audio signal is typically transformed into a 2D “image” using Digital Signal Processing (DSP) techniques, so that computer vision techniques can be used to classify the audio. Real world examples of this include audio or speech recognition, and keyword spotting.
This guide will introduce you to the DSP techniques used in these systems, using Python and NumPy code examples. Once you are familar with these DSP techniques, we will go through how to use Arm’s CMSIS-DSP library in Python to perform the same operations using fixed-point math.
Finally, we will demonstrate how to create an equivalent pipeline in C so that it can be deployed to an Arm Cortex-M based microcontroller (MCU). Using fixed-point math on Arm Cortex-M based systems can drastically reduce memory usage while improving run-time performance!
A Jupyter Notebook version of this guide can also be found on GitHub, which can also be accessed using Google Colab.
Audio Spectrograms
Audio Spectrograms in Python
Audio Spectrograms with CMSIS-DSP
Spectrogram examples for different audio sounds
Porting CMSIS-DSP code from Python to C
Conclusion
Throughout this notebook we will be developing an audio spectrogram pipeline in both NumPy and CMSIS-DSP. This is a common preprocessing pipeline used with audio data. The raw audio samples are converted into a visual representation of the audio signal which shows frequency changes over time. It is most often represented as a 2D image.
For a more in-depth explanation of spectrograms, you can take a look at the TensorFlow tutorial on audio recognition.
To convert an audio signal into the frequency domain, a Fast Fourier Transform (FFT) is used. This converts an audio signal represented by amplitude over time to the frequency domain, so you can understand the frequencies contained in the signal . However, when an FFT is performed on the entire audio signal, you will lose the time domain information for that audio signal, to counteract this we break the audio down into separate segments.
Creating a spectrogram can be broken down into the following steps:
An audio signal is broken down into shorter segments called "windows". This means when we later apply the FFT, we still keep some time information, because we know at what point in time each window occured. When splitting an audio signal into windows, it is common to have some overlap between windows to help keep the output more continuous.A windowing function is then applied to each window. Breaking our signal into windows that overlap will leave us with discontinuities in the audio signal which will not accurately represent the real-world audio signal. To reduce this impact, the windowing function will smooth the edges of each window's amplitude by multiplying the window by a smooth symmetric curve between 0 and 1.Each window is then converted into the frequency domain with an FFT. Because the FFT produces complex outputs, the absolute value is taken of this output to keep the amplitude information but remove the complex numbers.Finally a 2D array called the spectrogram is created of size (number of windows, FFT bins).
An audio signal is broken down into shorter segments called "windows". This means when we later apply the FFT, we still keep some time information, because we know at what point in time each window occured. When splitting an audio signal into windows, it is common to have some overlap between windows to help keep the output more continuous.
A windowing function is then applied to each window. Breaking our signal into windows that overlap will leave us with discontinuities in the audio signal which will not accurately represent the real-world audio signal. To reduce this impact, the windowing function will smooth the edges of each window's amplitude by multiplying the window by a smooth symmetric curve between 0 and 1.
Each window is then converted into the frequency domain with an FFT. Because the FFT produces complex outputs, the absolute value is taken of this output to keep the amplitude information but remove the complex numbers.
Finally a 2D array called the spectrogram is created of size (number of windows, FFT bins).
We've selected some example audio inputs along side their audio spectrogram representations for you below:
Now that you have an overview of what an audio spectrogram is, we’ll go through how to create one using Python and Numpy from a pre-recorded wave file.
You’ll need to install the NumPy, Matplotlib, SciPy and requests Python modules using pip :
pip install numpy scipy matplotlib requests
We’ll be using a “door knocking” .wav file from the "ESC-50: Dataset for Environmental Sound Classification" GitHub repo as the input signal for the DSP pipeline in the notebook. You can download the .wavfile from GitHub using the requests library from Python:
SciPy’s scipy.io.wavfile.read(...) API can now be used to read the audio samples from the .wav file.
The .wav file has a sampling rate of 44.1 kHz, which means each 1 second of audio is represented by 44,100 samples. Since the 44.1 kHz sampling rate contains more resolution than we need, we will use SciPy's scipy.signal.resample(...) API to lower the sampling rate of the audio data from 44.1 kHz to 16 kHz to reduce the number of samples used in the analysis.
Now that we have the audio samples read from the .wav file, let's plot the audio signal over time using Matplotlib:
You can also download and listen to the 1–103995-A-30.wav file on you computer.
Based on the graph of the entire audio signal, and playing the sound, we know there is about 1 second of sound at the start followed by silence. Let’s zoom in and only plot the first 1 second of the audio signal over time using Matplotlib to take a closer look:
Now we can split up the audio samples into overlapping windows. Typically a power of 2, such as 16, 32, 64, 128, 256, 512, 1024 or 2048 is selected as the window size. We’ll use a window size of 256 and step over the audio signal 128 samples at the time.
The current window is highlighted in green on top of the entire signal in blue. You can see the first window is from 0 to 256, the second window is from 128 to 384 (128 + 256), the third window is from 256 to 512, and the fourth window is from 384 to 640.
The total number of windows for the .wav file can be calculated as follows:
number_of_windows = (len(audio_samples) - window_size) // step_size
When processing continuous signals, like audio, a window function is applied before applying the FFT transform, so that the edges of the slices of audio are smoother. A windowing function is a function which (usually) ranges from (0–1) and can be applied to a window as follows:
y = window_function × window
One of the most common windowing functions (and the function used in this example) is the Hanning Window. NumPy has a built-in method to calculate a Hanning Windowfor any window length:
import numpy as nphanning_window = np.hanning(window_size)
Let’s plot out the Hanning Window using Matplotlib:
Now the Hanning Window can be applied to the first audio signal window using:
window_1 = audio_samples[0:window_size]processed_window_1 = hanning_window * window_1
We can then plot the first window and processed window using Matplotlib
You can see the amplitude of the processed window never exceeds the value of the Hanning Window throughout the window period.
Let’s extend our graph to loop through the first few windows.
For each processed window, the FFT can be calculated using NumPy’s np.fft.rfft(...) API and take the FFT's absolute value using np.absolute(...):
# Calculate the FFT.fft_1 = np.fft.rfft(processed_window_1)# Take the absolute value.fft_bins_1 = np.absolute(fft_1)
We can then plot the FFT bins using Matplotlib.
Let’s plot the FFT of the first 4 windows over time so we can see how the audio signal’s frequency content changes over time:
Instead of having separate FFT graphs for each window period over time, we can create a single 2D graph of the audio signal frequency domain over time as a heatmap, also known as a spectrogram! This spectrogram shows time on the x-axis, frequency on the y-axis and finally amplitude with colour.
Let’s loop through all the samples with an overlapping step and window size to calculate the signals frequency over time to display as a spectrogram. Here we calculate the number of windows in 1 second worth of audio.
Now that we’ve calculated the audio signal’s spectrogram, let’s plot it:
From the spectrogram plot we can clearly see 5 areas of interest for each knock, with frequencies below 4000 Hz being more prevalent.
Let’s zoom into the bottom half of the spectrogram and have a closer look:
Let’s have one more plot, but add the signal in the time domain above the spectrogram. Doing this, we can see the correlation between the signal’s amplitude and frequency counts both against time.
Let’s recap what we’ve accomplished so far:
Used the requests library to download a .wav file of a door knocking sound from GitHub.Used the SciPy library to read in the audio samples from a .wav file at 44.1 kHz and resampled them to have a 16 kHz sample rate.Understood what windowing is and why it is needed along with window functions like the Hanning Window.Learned how to create an FFT of an audio signal over a specific window period.Created a spectrogram representation of the audio signal by striding over the audio signal over time, applying a Hanning Window, and using the FFT function for each window period.
Used the requests library to download a .wav file of a door knocking sound from GitHub.
Used the SciPy library to read in the audio samples from a .wav file at 44.1 kHz and resampled them to have a 16 kHz sample rate.
Understood what windowing is and why it is needed along with window functions like the Hanning Window.
Learned how to create an FFT of an audio signal over a specific window period.
Created a spectrogram representation of the audio signal by striding over the audio signal over time, applying a Hanning Window, and using the FFT function for each window period.
Now that you understand how an audio spectrogram can be created using NumPy, we can do the same thing using the CMSIS-DSP Python Wrapper with fixed-point math.
Unlike floating-point operations, fixed-point math numbers represent real number with less precision as they have a fixed number of bits to represent a number before and after the decimal point. For example, if we have 16-bit number in 1.15 fixed-point format, the 16-bit value can represent a value between -1 and 1, with a fixed precision of 1 / 2^15 = 0.000030517578125.
Arm Cortex-M processors are faster at integer based calculations than floating-point operations. Arm Cortex-M4, Cortex-M7, and latter processors include support of Single Instruction Multiple Data (SIMD) instructions, which allows them to perform the same instruction on multiple values at a time. The CMSIS-DSP library leverages these SIMD instructions for more performant DSP processing.
Arm’s CMSIS-DSP library contains highly optimized implementations for common DSP operations for Arm Cortex-M processors.
Generally, you use the CMSIS-DSP library using C or C++ on the Arm Cortex-M device. However, the CMSIS-DSP team has created a Python Wrapper to leverage these DSP functions from with in Python.
The library contains both highly optimized DSP function implementations for Arm Cortex-M processors and generic C implementations for non-Arm processors. If you are using the Arm CMSIS-DSP Python wrapper on your PC (which is likely x86 based), the functions will behave the same as they do on Arm processors from an input and output perspective, however you might not see and increase in runtime performance.
Let’s go ahead and install the CMSIS-DSP Python Wrapper:
pip install git+https://github.com/ARM-software/CMSIS_5.git@5.8.0#egg=CMSISDSP\&subdirectory=CMSIS/DSP/PythonWrapper
We can use the arm_float_to_q15(..,) function from CMSIS-DSP to convert floating-point values to 16-bit fixed-point (Q15) values:
from cmsisdsp import arm_float_to_q15audio_samples_q15 = arm_float_to_q15(audio_samples)
Now let’s graph the fixed-point values along side the orignal NumPy values:
You can see the signal has a similiar shape to the one created with NumPy, however the values on the Y-axis range from -32768 to 32767 rather than 0 to 1.
CMSIS-DSP does not have a built-in function for creating a Hanning Window for a specific window size. However, we can levergage the built-in arm_cos_f32(...) and arm_float_to_q15(...) API's to create a fixed-point Hanning Window. Let's do this using the formula for the Hann Function:
Now let’s plot the Q15 Hanning Window along side the NumPy Hanning Window using Matplotlib:
You can see it has a similiar shape to the one created with NumPy, however the values range from 0 to 32767 rather than 0 to 1.
To apply the Hanning Window as we did in NumPy, we can use the arm_mult_q15(...)multiplication function to multiply two vectors of equal length together.
Lets now take a look at the first window after applying the Hanning Window in q15 format:
Let’s graph it side by side with the NumPy values:
CMSIS-DSP provides many FFT functions for various data types: q15, q31 and f32. In this example, we will be using the Real FFT functions. To use CMSIS-DSP’s Q15 RFFT function, we first need to create an arm_rfft_instance_q15 instance and initialize it with the arm_rfft_init_q15(...) function. After that, the process is similar to NumPy, you can use the arm_rfft_q15(...) function to calculate the FFT and arm_cmplx_mag_q15(...) to calculate the magitude of the FFT.
We can now graph the Q15 fixed-point values with Matplotlib:
Now let’s convert the fft_bins_1_q15 Q15 values to be be floating-point value, so we can compare the value with the NumPy calculated value.
The output format of the arm_rfft_q15(...) function when an RFFT length of 256 is used, will be 9.7 (instead of 1.15). The documentation for the arm_cmplx_mag_q15(...)function explains input values are in 1.15 format and outputs will be in 2.14 format. So, if we pass an 9.7 format number into arm_cmplx_mag_q15(...), the output will have a format of 10.6 - which mean's it's value will range from -512 to 511.
We can convert the 10.6 number into a floating point value by converting it from a Q15 to a float using the arm_q15_to_float(...) function and multiplying it by 512 (2^9):
from cmsisdsp import arm_q15_to_float# Let's rescale them and compare the twofft_bins_1_q15_scaled = arm_q15_to_float(fft_bins_1_q15) * 512
We can overlay them to see how close they are:
As we can see from above, the two pipelines produce very visually similar features.
Although some information is lost using fixed-point functions, we managed to calculate a similar shape and show peaks at the same frequencies. The frequencies that the fixed-point functions did not manage to calculate (to the right of 2000), all have a really low amplitude showing the information lost is minimal.
Finally, we need to put all this together as we did with NumPy.
Now let’s graph the both the input signal and Q15 Spectrogram with Matplotlib:
And finally compare Q15 spectrogram with the NumPy version:
As we have seen, the CMSIS-DSP pipeline is similar to the NumPy pipeline. Let’s recap the what we’ve done to create it:
Converted the audio samples into Q15 format using the arm_float_to_q15 function.Calculated the Hanning Window using the Hann Function formula using the arm_cos_f32and arm_float_to_q15 functions.Learned how to multiply to vectors of the same length using the arm_mult_q15 function.Learned how to calculate a Q15 FFT in CMSIS-DSP using the arm_rfft_instance_q15, arm_rfft_init_q15, and arm_rfft_q15 functions.Created a spectrogram representation of the audio signal by striding over the audio signal over time, applying a Hanning Window, and using the FFT function for each window period.
Converted the audio samples into Q15 format using the arm_float_to_q15 function.
Calculated the Hanning Window using the Hann Function formula using the arm_cos_f32and arm_float_to_q15 functions.
Learned how to multiply to vectors of the same length using the arm_mult_q15 function.
Learned how to calculate a Q15 FFT in CMSIS-DSP using the arm_rfft_instance_q15, arm_rfft_init_q15, and arm_rfft_q15 functions.
Created a spectrogram representation of the audio signal by striding over the audio signal over time, applying a Hanning Window, and using the FFT function for each window period.
Let’s now take a look at a few different spectrograms for different audio files in the ESC-50 repository.
Now that we have a general understanding of how to use the various CMSIS-DSP API’s in Python from the last section, we can start to map these to C to run on our Cortex-M based system.
The first step will be to add an #include <arm_math.h> to the top of your .c file:
#include <arm_math.h>
Then we can define some constant variables for the window and step size:
const int WINDOW_SIZE = 256;const int STEP_SIZE   = 128;
A global array variable can be declared to store the input signal:
q15_t input_q15[WINDOW_SIZE];
The input signal to create the spectrum for can be stored in the input_q15 variable.
For testing purposes, we’ll set it to be a fixed sine wave with a frequency of 440 Hz sampled at 16 kHz.
for (int i = 0; i < WINDOW_SIZE; i++) {    float32_t f = sin((2 * PI * 440) / 16000 * i);        arm_float_to_q15(&f, &input_q15[i], 1);}
For the Hanning Window function you must declare a global array variable to store the window function values:
q15_t hanning_window_q15[WINDOW_SIZE];
Then we can create a C function that initializes the hanning_window_q15 array at runtime:
void hanning_window_init_q15(q15_t* hanning_window_q15, size_t size) {    for (size_t i = 0; i < size; i++) {        // calculate the Hanning Window value for i as a float32_t        float32_t f = 0.5 * (1.0 - arm_cos_f32(2 * PI * i / size ));                // convert value for index i from float32_t to q15_t and        // store in window at position i        arm_float_to_q15(&f, &hanning_window_q15[i], 1);    }}
We will need another variable to store the value of the applied Hanning Window on the Input Signal:
q15_t processed_window_q15[WINDOW_SIZE];
Now we can apply the Hanning Window on the Input Signal:
// equivalent to://   processed_window_q15 = input_q15 * hanning_window_q15arm_mult_q15(input_q15, hanning_window_q15, processed_window_q15,             WINDOW_SIZE);
Variables
We’ll need global variables for the FFT instance and outputs:
arm_rfft_instance_q15 S_q15;// this is twice the size because each FFT output has a real and// imaginary partq15_t fft_q15[WINDOW_SIZE * 2];// this is half the size of WINDOW_SIZE becase we just need the// magnitude from the first half of the FFT outputq15_t fft_mag_q15[WINDOW_SIZE / 2];
Initialization
We need to call arm_rfft_init_q15 to initialize the Q15 RFFT instance:
arm_rfft_init_q15(&S_q15, WINDOW_SIZE, 0, 1);
Performing the FFT
Performing the FFT and calculating the FFT’s outputs magnitude can now be done with:
arm_rfft_q15(&S_q15, processed_window_q15, fft_q15);arm_cmplx_mag_q15(fft_q15, fft_mag_q15, WINDOW_SIZE / 2);
Now let’s benchmark the Arduino sketch from above on the following Arduino or Arduino compatible boards:
Arduino Nano 33 IoT
Arduino Nano 33 BLE
Teensy 4.0
We’ve created an equivalent sketch that uses 32-bit floating-point instead of fixed-point as a comparison against our fixed-point sketch above. Both sketches can be found on GitHub:
cmsis_dsp_fft_q15.ino
cmsis_dsp_fft_f32.ino
RAM Usage
We should expect to see the buffers we need for the DSP operations to take half as much space when we use 16-bit Q15 instead of 32-bit floating-point.
Our current buffers include:
So with a WINDOW_SIZE = 256 we expect:
F32: 22 x 256 = 5632 bytes
Q15: 11 x 256 = 2816 bytes
This illustrates that using fixed-point with Q15 requires half the RAM compared to using floating-point with F32.
If we compile our sketches we get the following metrics for “Global variables use”.
The values from our Arduino match exactly to the amount of RAM we expected to save.
Run-time speed
Now let’s upload both benchmarking sketches to the boards to see how long it takes to calculate the FFT bins for a single input window:
The Arduino Nano 33 IoT board which is based on an Arm Cortex-M0+ CPU and is not equipped with an FPU (Floating Point Unit) completes the pipline 11 times faster using the 16-bit fixed-point DSP pipeline instead of the 32-bit floating-point version. The Arduino Nano 33 BLE which is based on an Arm Cortex-M4F CPU and equipped with an FPU has a improvement of 37%. Finally, the Teensy 4.0 which is based on an Arm Cortex-M7 CPU which is equipped with an FPU has a improvement of 26%.
This guide has given an overview of a DSP pipeline to convert an audio signal into a spectrogram, which can then be combined with computer vision techniques to classify audio signals. We’ve explained what a spectrogram is and how to create one from a .wav file in Python and NumPy using the following DSP operatons: Windowing, the Hanning Window and FFT’s. After this, you’ve learned about fixed-point math and how the CMSIS-DSP Python Wrapper can be used to create an equivalent spectrogram using 16-bit fixed-point operations with minimal loss in accuracy. We then covered how to port the CMSIS-DSP pipeline from Python to C to run on an Arm Cortex-M based device.
In our benchmarks, the global buffers required by the DSP pipeline required 50% less memory when using fixed-point 16-bit operations instead of 32-bit floating-point operations. The Cortex-M0+ (with no FPU) had a 11x speed increase when using fixed-point 16-bit operations instead of floating-point. The Cortex-M4F and Cortex-M7F systems with built-in FPU’s saw a 37% and 26% increase in speed when using fixed-point 16-bit operations instead of floating-point. These are impressive improvements. Using fixed-point also saves a conversion step, audio input values are (often) stored as 16-bit values when they are read from analog microphones (using an ADC) or digital microphones.
For more blogs, guides, resources, videos and notebooks from the Arm Software Developer’s team please visit:
@ArmSoftwareDev on Twitter
The Arm Software Developers YouTube channel
@ArmDeveloperEcosystem on GitHub
Up-skill and get hands-on experience using tinyML at the upcoming Arm DevSummit, a 3-day virtual event between October 19–21. The event includes workshops on tinyML computer vision for real-world embedded devices and building large vocabulary voice control with Arm Cortex-M based MCUs. We hope to see you there!