Repository: scipy/scipy

Issue:
signal.decimate() introduces nasty phase-shift
The `decimate()` function by default adds a strong phase shift to the data, apparently because it used a single pass (low pass filter and not `scipy.signal.filtfilt()` 

The problem has been reported earlier[1][2], and the suggested workaround is to clone-and-adapt the code of the function.

I tried this on my sample data and got satisfactory results:
![decimatephaseshift](https://cloud.githubusercontent.com/assets/501585/7927242/c1a5ad14-08dd-11e5-9f60-807d08175eec.png)

May I provide a PR to modify `decimate()` to accept a new boolean _kw-arg_ `filtfilt` to do the above?
And possibly another one, `filtfilt_kws` ??

[1] http://mail.scipy.org/pipermail/scipy-user/2011-April/029063.html
[2] http://scipy-user.10969.n7.nabble.com/SciPy-User-Decimate-and-filtfilt-td20183.html



Discussion:
+1 from me

While `decimate` is up for discussion, my two cents:

The current IIR filter chosen is a type-1 Chebyshev, but with only 0.05dB allowed passband ripple. At that point, I feel it may be more advantageous use a type-2 Cheby filter, which naturally has a flat passband, and has a notch in the stopband which can be aligned with the nyquist frequency. 

Alternatively, if we want to make the filtering more user-controllable, this seems like a natural place to use something the  `[filt]ord` functions, (like `cheb2ord`, `ellipord`, etc.) to let the user specify how steep they want the cutoff / how flat they want the passband / what kind of filter they want to use (along with some sensible defaults). 

Thanks for bringing this up, I was unaware decimate introduced this phase shift.

About the keyword. I don't think there are (or will be!) any other options than `lfilter` or `filtfilt` and therefore I think a better name would be  `zero_phase=True` for `filtfilt`.

`filtfilt` has some keywords, and you might want to pass those as well. One solution would indeed be `filtfilt_kws`.

Another would be using something like `mode='lfilter'` or `mode=filtfilt` or `mode=some_func`. Why? While I don't expect one would use any other function than `lfilter` or `filtfilt` this method would allow you so and this way you could also add keywords through the use of `functools.partial`.

I agree, the power of `partial()` is frequently under-appreciated.

But in this case i would prefer the 1st suggestion, the a `zero_phase` +`filtfilt_kws`, because it looks simpler; for more complex situations, decimate is a small and you can easily clone and adapt.

The problem is that a 2-pass filtering needs lower-order filter-coefficients.
In my case i have found the following defaults for the `n` parameter to work:

| filter-type | # of passes | filter-order |
| --- | --- | --- |
| `fir` | `single-pass` | 30 |
| `iir` | `single-pass` | 8 |
| `fir` | `2-pass` | 26 |
| `iir` | `2-pass` | 6 |

But i would expect more trials on this.

@e-q the `cheby1` with 0.05 dB passband seems to have been chosen to match Matlab: http://nl.mathworks.com/help/signal/ref/decimate.html

Making this more configurable without complicating the API with even more keywords looks difficult. Instead of more strings or a callable, the simplest might be to just accept one of the `lti` subclasses or a tuple `(b, a)`.

@ankostis a PR adding `filtfilt` support sounds good. `zero_phase` makese sense. Not 100% sure about `filtfilt_kws`, that makes the API a bit complex.

The problem with this function is that it's basically a two-liner:

```
b, a = construct_filter(n, q, kwds)
res = lowpass_filter(b, a, x, other_kwds)
```

Adding lots of keywords to `decimate` to tweak the behavior of `construct_filter` or `lowpass_filter` quickly makes `decimate` less usable than a reimplementation by hand.

Something like `decimate(x, q, filt, axis=-1, zero_phase=False)` where `filt` is an instance of a subclass of `lti`? 
This would be backwards incompatible though.

Unless, the `zero_phase` keyword would be added, and `ftype` could be an instance of a subclass of `lti` besides the current `iir` and `fir` options.

Breaking backwards compat isn't needed I think. Adding `zero_phase=False` at the end, and making `ftype` accept `signal.TransferFunction` is what I meant.

I added the `zero_phase` keyword as well as the ability for `ftype` to accept an instance of `lti`.
Please have a look at https://gist.github.com/ba60084570d05c91e9bd

You see two versions, the one where I use `b, a` and the other where I use an instance of `system` internally. While right now it doesn't matter at all which is used, I think for in the future it's best to already use the system representation. As soon as we then have a `SecondOrderSections` class and more functions support `sos` it's rather straight forward to convert.

@FRidh, I've used the code you posted for a little bit, and it seems fine to me. Why not open a PR for it?


Resolving patch:
diff --git a/scipy/signal/signaltools.py b/scipy/signal/signaltools.py
--- a/scipy/signal/signaltools.py
+++ b/scipy/signal/signaltools.py
@@ -7,7 +7,7 @@
 import threading
 import sys
 
-from . import sigtools
+from . import sigtools, lti
 from ._upfirdn import _UpFIRDn, _output_len
 from scipy._lib.six import callable
 from scipy._lib._version import NumpyVersion
@@ -23,8 +23,8 @@
 from scipy.special import factorial
 from .windows import get_window
 from ._arraytools import axis_slice, axis_reverse, odd_ext, even_ext, const_ext
-from scipy.signal.filter_design import cheby1
-from scipy.signal.fir_filter_design import firwin
+from .filter_design import cheby1
+from .fir_filter_design import firwin
 
 if sys.version_info.major >= 3 and sys.version_info.minor >= 5:
     from math import gcd
@@ -2864,7 +2864,7 @@ def sosfilt(sos, x, axis=-1, zi=None):
     return out
 
 
-def decimate(x, q, n=None, ftype='iir', axis=-1):
+def decimate(x, q, n=None, ftype='iir', axis=-1, zero_phase=None):
     """
     Downsample the signal by using a filter.
 
@@ -2876,13 +2876,23 @@ def decimate(x, q, n=None, ftype='iir', axis=-1):
     x : ndarray
         The signal to be downsampled, as an N-dimensional array.
     q : int
-        The downsampling factor.
+        The downsampling factor. For downsampling factors higher than 13, it is
+        recommended to call `decimate` multiple times.
     n : int, optional
-        The order of the filter (1 less than the length for 'fir').
-    ftype : str {'iir', 'fir'}, optional
-        The type of the lowpass filter.
+        The order of the filter (1 less than the length for 'fir'). Defaults to
+        8 for 'iir' and 30 for 'fir'.
+    ftype : str {'iir', 'fir'} or ``lti`` instance, optional
+        If 'iir' or 'fir', specifies the type of lowpass filter. If an instance
+        of an `lti` object, uses that object to filter before downsampling.
     axis : int, optional
         The axis along which to decimate.
+    zero_phase : bool, optional
+        Prevent phase shift by filtering with `filtfilt` instead of `lfilter`
+        when `ftype` is not 'fir'.  A value of `True` is recommended, since a
+        phase shift is generally not desired. Using `None` defaults to `False`
+        for backwards compatibility.
+
+        .. versionadded:: 0.18.0
 
     Returns
     -------
@@ -2893,24 +2903,51 @@ def decimate(x, q, n=None, ftype='iir', axis=-1):
     --------
     resample
     resample_poly
+
+    Notes
+    -----
+    The ``zero_phase`` keyword was added in 0.18.0.
+    The possibility to use instances of ``lti`` as ``ftype`` was added in
+    0.18.0.
     """
 
     if not isinstance(q, int):
         raise TypeError("q must be an integer")
 
-    if n is None:
-        if ftype == 'fir':
-            n = 30
-        else:
-            n = 8
+    if n is not None and not isinstance(n, int):
+        raise TypeError("n must be an integer")
 
     if ftype == 'fir':
-        b = firwin(n + 1, 1. / q, window='hamming')
-        a = 1.
-    else:
-        b, a = cheby1(n, 0.05, 0.8 / q)
+        if n is None:
+            n = 30
+        system = lti(firwin(n + 1, 1. / q, window='hamming'), 1.)
 
-    y = lfilter(b, a, x, axis=axis)
+    elif ftype == 'iir':
+        if n is None:
+            n = 8
+        system = lti(*cheby1(n, 0.05, 0.8 / q))
+    elif isinstance(ftype, lti):
+        system = ftype
+    else:
+        raise ValueError('invalid ftype')
+
+    if zero_phase is None:
+        warnings.warn(" Note: Decimate's zero_phase keyword argument will "
+                      "default to True in a future release.  Until then, "
+                      "decimate defaults to 'one-way filtering for backwards "
+                      "compatibility. Ideally, always set this argument "
+                      "explicitly.", FutureWarning)
+        zero_phase = False
+
+    if zero_phase and ftype == 'fir':
+        warnings.warn('zero_phase is not implemented for FIR downsampling, '
+                      'using zero_phase=False.')
+        zero_phase = False
+
+    if zero_phase:
+        y = filtfilt(system.num, system.den, x, axis=axis)
+    else:
+        y = lfilter(system.num, system.den, x, axis=axis)
 
     sl = [slice(None)] * y.ndim
     sl[axis] = slice(None, None, q)


Test patch:
diff --git a/scipy/signal/tests/test_signaltools.py b/scipy/signal/tests/test_signaltools.py
--- a/scipy/signal/tests/test_signaltools.py
+++ b/scipy/signal/tests/test_signaltools.py
@@ -1511,18 +1511,85 @@ def test_filtfilt_gust():
 
 
 class TestDecimate(TestCase):
+    def test_bad_args(self):
+        x = np.arange(12)
+        assert_raises(TypeError, signal.decimate, x, q=0.5, n=1)
+        assert_raises(TypeError, signal.decimate, x, q=2, n=0.5)
 
-    def test_basic(self):
-        x = np.arange(6)
-        assert_array_equal(signal.decimate(x, 2, n=1).round(), x[::2])
+    def test_basic_IIR(self):
+        x = np.arange(12)
+        y = signal.decimate(x, 2, n=1, ftype='iir', zero_phase=False).round()
+        assert_array_equal(y, x[::2])
+
+    def test_basic_FIR(self):
+        x = np.arange(12)
+        y = signal.decimate(x, 2, n=1, ftype='fir', zero_phase=False).round()
+        assert_array_equal(y, x[::2])
 
     def test_shape(self):
         # Regression test for ticket #1480.
-        z = np.zeros((10, 10))
-        d0 = signal.decimate(z, 2, axis=0)
-        assert_equal(d0.shape, (5, 10))
-        d1 = signal.decimate(z, 2, axis=1)
-        assert_equal(d1.shape, (10, 5))
+        z = np.zeros((30, 30))
+        d0 = signal.decimate(z, 2, axis=0, zero_phase=False)
+        assert_equal(d0.shape, (15, 30))
+        d1 = signal.decimate(z, 2, axis=1, zero_phase=False)
+        assert_equal(d1.shape, (30, 15))
+
+    def test_phaseshift_FIR(self):
+        self._test_phaseshift(method='fir', zero_phase=False)
+
+    def test_phaseshift_IIR(self):
+        self._test_phaseshift(method='iir', zero_phase=False)
+
+    def test_zero_phase_IIR(self):
+        self._test_phaseshift(method='iir', zero_phase=True)
+
+    def _test_phaseshift(self, method, zero_phase):
+        rate = 120
+        rates_to = [15, 20, 30, 40]  # q = 8, 6, 4, 3
+
+        t_tot = int(100)  # Need to let antialiasing filters settle
+        t = np.arange(rate*t_tot+1) / float(rate)
+
+        # Sinusoids at 0.8*nyquist, windowed to avoid edge artifacts
+        freqs = np.array(rates_to) * 0.8 / 2
+        d = (np.exp(1j * 2 * np.pi * freqs[:, np.newaxis] * t)
+             * signal.tukey(t.size, 0.1))
+
+        for rate_to in rates_to:
+            q = rate // rate_to
+            t_to = np.arange(rate_to*t_tot+1) / float(rate_to)
+            d_tos = (np.exp(1j * 2 * np.pi * freqs[:, np.newaxis] * t_to)
+                     * signal.tukey(t_to.size, 0.1))
+
+            # Set up downsampling filters, match v0.17 defaults
+            if method == 'fir':
+                n = 30
+                system = signal.lti(signal.firwin(n + 1, 1. / q,
+                                                  window='hamming'), 1.)
+            elif method == 'iir':
+                n = 8
+                wc = 0.8*np.pi/q
+                system = signal.lti(*signal.cheby1(n, 0.05, wc/np.pi))
+
+            # Calculate expected phase response, as unit complex vector
+            if zero_phase is False:
+                _, h_resps = signal.freqz(system.num, system.den,
+                                          freqs/rate*2*np.pi)
+                h_resps /= np.abs(h_resps)
+            else:
+                h_resps = np.ones_like(freqs)
+
+            y_resamps = signal.decimate(d.real, q, n, ftype=system,
+                                        zero_phase=zero_phase)
+
+            # Get phase from complex inner product, like CSD
+            h_resamps = np.sum(d_tos.conj() * y_resamps, axis=-1)
+            h_resamps /= np.abs(h_resamps)
+            subnyq = freqs < 0.5*rate_to
+
+            # Complex vectors should be aligned, only compare below nyquist
+            assert_allclose(np.angle(h_resps.conj()*h_resamps)[subnyq], 0,
+                            atol=1e-3, rtol=1e-3)
 
 
 class TestHilbert(object):