Noise cancellation with Python and Fourier Transform
By: Piero Paialunga
Tags: Machine Learning, Noise, Time Series Analysis, Python, Signal

Noise cancellation with Python and Fourier Transform

Here’s how to use a very simple tool like Fourier Transform to obtain efficient noise cancellation, with few lines of code. Piero Paialunga Sep 5·6 min read

Modelling the noise is a really hard task. And I’m not even trying to argue with that. As a physicist, I deal with noise all the times, and it is not easy to capture it, model it, or deal with it in general.

For this reason, a lot of complex, smart, and efficient methods have been developed during the years. With the technological progresses that computer have experienced during the years, great performances of Machine (Deep) Learning de-noising algorithms have been obtained. While these methods permit to obtain incredible results, sometimes very simple approaches based on extremely reasonable and general considerations can be used to solve the noise cancellation problem with excellent results.

An important example of this concept is the Fourier de-noising approach.

1. The theory

Let’s say you have a signal that, mathematically speaking, can be considered as a function from a real space to a real number:

I generated this image using this tool

The idea of the Fourier transform is to study this signal in another domain.

More specifically, the domain that you use is the frequency domain, thus obtaining:

I generated this image using this tool

So what does it mean? And why is it so important?

Well, the idea is that you can decompose your signal as a discrete sum (or a continuous sum i.e. an integral) of sines and cosines, considered with their specific amplitudes.

Let’s make it easier. Let’s pretend that you have this signal:

Image generated by me using Python

That means that your signal is y:

I generated this image using this tool

Now, as this signal is just a sine, we will have that, in the Fourier space, it looks like this:

Image generated by me using Python

And we have 1 as the frequency of the sine is 1 (think of the signal as y=sin(omega x).

This is obtained with a reversible function that is the fast Fourier transform. You can easily go back to the original function using the inverse fast Fourier transform.

So why are we talking about noise cancellation?

A safe (and general) assumption is that the noise can survive at all the frequencies, while your signal is limited in the frequency spectrum (namely band-limited) and has only certain specific non-null frequencies that characterize it.

Let’s say that you have a noisy sine function like this one:

Image generated by me using Python

The frequency spectrum is this one:

Image generated by me using Python

And while you can see the peak at omega=1, everything else is just noise.

A general assumption that has to be done is that the signal and the noise are non-correlated, and that, even if your signal is noisy, the “non-noise” part of the signal is dominant.

So the main idea is to find the real signal frequencies and to obtain a reconstructed signal by using the important frequencies of the signal only. The “non relevant frequencies”, which are going to be found using the correlation values, will be set to 0.

I know it all sounds confusing, but I really think that it will be far more clear after a practical example. So follow me!

2. The Dataset

In order to do all the analysis that we’ll make, we’ll need these libraries:

I downloaded the data from here. It is a time-series, that is the perfect example of what we need:

We can decide whatever feature we want. I analyzed the “Close” one for no specific reason, so feel free to repeat the same process on another feature!

3. Data Pre-Processing

The first main problem we see is that we don’t have an equally sampled time space. For example, we have the first date that is the first day of April (2015–04–01), the second one that is the second day of April (2015–04–02) and the third one that is the sixth day (2015–04–06).

We can solve this problem by considering a continuous time space and adding the values related to the closest day available.

In practice you do this with these lines of code and, as you can see, the two time-series (the real one and the artificial) are essentially the same:

The second problem is that Fourier analysis works well only for stationary data and we can clearly see that this time series is increasing during the years.

We are fixing this using a bit of Machine Learning. In particular, we will use Polynomial Regression to find the best-fit polynomial function that fits the data. Then we will remove this line and obtain the stationary time-series:

Here is how we’ll do it:

4. Denoising Algorithm

Let’s get the Fourier transform and plot the amplitude:

Ok, so the idea is to filter it. We are cutting out all the frequencies below a certain level. This level is set by using the maximum amplitude as a reference value. For example, you can drop all the frequencies below 0.7 times the maximum amplitude.

This is an example of how it works.

If we inverse transform each filtered Fourier transforms we get the following results:

And here’s all the functions that you can use the filter the data based on a certain threshold:

So the thing that we can do now to get the optimal threshold is to look at the correlation between the original signal and the difference between the original signal and the reconstructed one. If the correlation is high, that means that we are still missing something from the original signal. Otherwise, as the signal should not be correlated to the noise, we can assume that we are just deleting the noise, that is what we want to do.

Here is what we can get:

And here is the correlation (correlation value and p-value) at different value of threshold:

Thus, we find that the optimal threshold is 0.004 times the Maximum Value!

Here is the result!

They seem very similar, and in fact, we’ve been really conservative! That means that we really do not want to cancel important information of the signal, even if it means that we may not delete all the noise.

Another thing we can do is to relax this threshold. This will surely reduce more noise, but it will probably cut something important out of the signal as well.

Final Considerations

This blog post wants to show that sometimes good old fashioned methods could still help you solving problems without using crazy fast GPU and advanced Deep Learning algorithms.

Of course you may have problems with more specific kind of noise and it could be way harder than the one explained in this post. Anyway, in a general case when you don’t really know the problem and you are just exploring it, Fourier denoising method could be a really good starting point for your research!

I really hoped you liked it! Feel free to write here: piero.paialunga@hotmail.com

Thank you so much and see you soon!