Repository: nilearn/nilearn

Issue:
Filtering looks broken
Our usage of butterworth looks like it is not doing what is should.



Discussion:
Related to #374 ?

Can you be more specific?

Sorry, I wrote this issue live during the course at Leipzig, in front of the audience. I dug a bit more, and the following seems really wrong:
</pre>
import numpy as np
from nilearn.signal import butterworth
from matplotlib import pyplot as plt

t = np.linspace(0, 10, 2000)
y = np.cos(t)

plt.clf()
plt.plot(y)
plt.plot(butterworth(y[:, np.newaxis], sampling_rate=1., low_pass=10000.,
         ).squeeze())

plt.show()
</pre>

Even  though I am putting a very high cut off frequency for the low_pass filter, I am getting the signal entirely cut. Even stranger, if I change the low_pass to a high_pass, I am getting the opposite: no cut at all. It seems to me that there is something deeply broken here.

On 04/03/2015 21:39, Gael Varoquaux wrote:

> plt.plot(butterworth(y[:, np.newaxis], sampling_rate=1., low_pass=10000.,
> ).squeeze())
> On numpy 1.9.1, this throws an error ("wn should be btw 0 and 1").

> On numpy 1.9.1, this throws an error ("wn should be btw 0 and 1").

Not on my box.

> On 04/03/2015 21:39, Gael Varoquaux wrote:
> 
> > plt.plot(butterworth(y[:, np.newaxis], sampling_rate=1., low_pass=10000.,
> > ).squeeze())
> > On numpy 1.9.1, this throws an error ("wn should be btw 0 and 1").

Yes, we have had this in the workshop also. It clearly seems to me that
this code is broken.

OK, I think that I got it. In fact, Wn (see scipy.signal.butter) must be in [0, 1] for a digital filter. The problem is that when the value is outside of the bounds, the filter seems to have an unexpected behavior.

You can see it easily: with `low_pass=0.5` you get the expected behavior (signal is the same, `Wn=1.`). But with `low_pass=0.501` you can see that the filter goes completely crazy. 

I still don't know why I don't get the warning associated with Wn value.

> Related to #374 ?

No. This issue has been clearly identified and we only need to agree on the behavior to adopt if a high pass filter is applied on data with a linear trend.

> OK, I think that I got it. In fact, Wn (see scipy.signal.butter) must
> be in [0, 1] for a digital filter. 

It seems that we should clip it (and maybe warn) for values too high or
too low.

:+1:

> On numpy 1.9.1, this throws an error ("wn should be btw 0 and 1").

I also get this error on Mac + numpy '1.9.2'


Resolving patch:
diff --git a/nilearn/signal.py b/nilearn/signal.py
--- a/nilearn/signal.py
+++ b/nilearn/signal.py
@@ -8,6 +8,7 @@
 # License: simplified BSD
 
 import distutils.version
+import warnings
 
 import numpy as np
 import scipy
@@ -148,6 +149,17 @@ def _detrend(signals, inplace=False, type="linear", n_batches=10):
     return signals
 
 
+def _check_wn(btype, freq, nyq):
+    wn = freq / float(nyq)
+    if wn > 1.:
+        warnings.warn('The frequency specified for the %s pass filter is '
+                'too high to be handled by a digital filter (superior to '
+                'nyquist frequency). It has been lowered to %.2f (nyquist '
+                'frequency).' % (btype, nyq))
+        wn = 1.
+    return wn
+
+
 def butterworth(signals, sampling_rate, low_pass=None, high_pass=None,
                 order=5, copy=False, save_memory=False):
     """ Apply a low-pass, high-pass or band-pass Butterworth filter
@@ -203,22 +215,21 @@ def butterworth(signals, sampling_rate, low_pass=None, high_pass=None,
 
     nyq = sampling_rate * 0.5
 
-    wn = None
-    if low_pass is not None:
-        lf = low_pass / nyq
-        btype = 'low'
-        wn = lf
-
+    critical_freq = []
     if high_pass is not None:
-        hf = high_pass / nyq
         btype = 'high'
-        wn = hf
+        critical_freq.append(_check_wn(btype, high_pass, nyq))
 
-    if low_pass is not None and high_pass is not None:
+    if low_pass is not None:
+        btype = 'low'
+        critical_freq.append(_check_wn(btype, low_pass, nyq))
+
+    if len(critical_freq) == 2:
         btype = 'band'
-        wn = [hf, lf]
+    else:
+        critical_freq = critical_freq[0]
 
-    b, a = signal.butter(order, wn, btype=btype)
+    b, a = signal.butter(order, critical_freq, btype=btype)
     if signals.ndim == 1:
         # 1D case
         output = signal.filtfilt(b, a, signals)


Test patch:
diff --git a/nilearn/tests/test_signal.py b/nilearn/tests/test_signal.py
--- a/nilearn/tests/test_signal.py
+++ b/nilearn/tests/test_signal.py
@@ -138,6 +138,16 @@ def test_butterworth():
                           copy=False)
     np.testing.assert_almost_equal(out1, data)
 
+    # Test nyquist frequency clipping, issue #482
+    out1 = nisignal.butterworth(data, sampling,
+                                 low_pass=50.,
+                                 copy=True)
+    out2 = nisignal.butterworth(data, sampling,
+                                 low_pass=80.,  # Greater than nyq frequency
+                                 copy=True)
+    np.testing.assert_almost_equal(out1, out2)
+
+
 
 def test_standardize():
     rand_gen = np.random.RandomState(0)