Repository: pandas-dev/pandas

Issue:
BUG: rolling_window yields unexpected results with win_type='triang'
Here's the example in the documentation, modified to have non-zero mean:

```
n = 100
ser = pandas.Series(randn(n)+10, index=pandas.date_range('1/1/2000', periods=n))
pandas.rolling_window(ser, 5, 'triang').plot()
pandas.rolling_window(ser, 5, 'boxcar').plot()
```

The rolling boxcar window is centered around 10, as expected.

The triang window is centered around 6.  That suggests that the weights in the window don't add up to 1.

Either that or my expectation of how it should work is wrong?



Discussion:
these are `scipy` dependent so

```
import scipy.signal as sig
sig.get_window(win_type, window)
```

yields the weights

Understood.  It looks like there is some logic in rolling_window that is
supposed to normalize the weights.  Maybe that is not correct?

On Mon, Jun 30, 2014 at 11:13 AM, jreback notifications@github.com wrote:

> these are scipy dependent so
> 
> import scipy.signal as sig
> sig.get_window(win_type, window)
> 
> yields the weights
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/issues/7618#issuecomment-47543889.

@AllenDowney certainly possible (not sure whether this should be normalized or not). Much of this I don't think is actually tested (as it relies on the weights from scipy). Want to put some test cases together? 

I don't have a pandas dev environment set up, but here's a simple case:

```
n = 10
ser = pandas.Series(np.ones(n))

mean = pandas.rolling_window(ser, 5, 'triang').mean()
np.testing.assert_approx_equal(mean, 1.0)

mean = pandas.rolling_window(ser, 5, 'gaussian', std=1.5).mean()
np.testing.assert_approx_equal(mean, 1.0)

mean = pandas.rolling_window(ser, 5, 'boxcar').mean()
np.testing.assert_approx_equal(mean, 1.0)
```

Currently only 'boxcar' is passing.

One theory: when rolling_window calls algos.roll_window, should it be
sending avg_wgt=True?

Allen

On Mon, Jun 30, 2014 at 11:48 AM, jreback notifications@github.com wrote:

> @AllenDowney https://github.com/AllenDowney certainly possible (not
> sure whether this should be normalized or not). Much of this I don't think
> is actually tested (as it relies on the weights from scipy). Want to put
> some test cases together?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/issues/7618#issuecomment-47548507.

```
In [10]: sig.get_window(('gaussian',1.5),5)
Out[10]: array([ 0.41111229,  0.8007374 ,  1.        ,  0.8007374 ,  0.41111229])

In [11]: sig.get_window('boxcar',5)
Out[11]: array([ 1.,  1.,  1.,  1.,  1.])

In [12]: sig.get_window('triang',5)
Out[12]: array([ 0.33333333,  0.66666667,  1.        ,  0.66666667,  0.33333333])
```

Do these need to be normalized?

Yes, I think so.

On Mon, Jun 30, 2014 at 1:12 PM, jreback notifications@github.com wrote:

> In [10]: sig.get_window(('gaussian',1.5),5)
> Out[10]: array([ 0.41111229,  0.8007374 ,  1.        ,  0.8007374 ,  0.41111229])
> 
> In [11]: sig.get_window('boxcar',5)
> Out[11]: array([ 1.,  1.,  1.,  1.,  1.])
> 
> In [12]: sig.get_window('triang',5)
> Out[12]: array([ 0.33333333,  0.66666667,  1.        ,  0.66666667,  0.33333333])
> 
> Do these need to be normalized?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/issues/7618#issuecomment-47558559.

so why is 'boxcar' right then?

Because the window is all 1s, it is already normalized.

On Mon, Jun 30, 2014 at 1:19 PM, jreback notifications@github.com wrote:

> so why is 'boxcar' right then?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/issues/7618#issuecomment-47559376.

Indeed it appears to be a normalization issue. The result is correct (i.e. it overlaps with the original signal) if you multiply it by the following number:

``` python
window / sum(sig.get_window('triang', window))
```

@marco-giancotti @AllenDowney 

either of you want to put some tests / fix in place for this?

ideally some comprehensive testing of all window types to validate this 

I might have time to do this next week, but would not be deeply offended if
someone else beat me to it.

Allen

On Thu, Aug 7, 2014 at 10:24 AM, jreback notifications@github.com wrote:

> @marco-giancotti https://github.com/marco-giancotti @AllenDowney
> https://github.com/AllenDowney
> 
> either of you want to put some tests / fix in place for this?
> 
> ideally some comprehensive testing of all window types to validate this
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/issues/7618#issuecomment-51478404.


Resolving patch:
diff --git a/pandas/util/print_versions.py b/pandas/util/print_versions.py
--- a/pandas/util/print_versions.py
+++ b/pandas/util/print_versions.py
@@ -68,7 +68,6 @@ def show_versions(as_json=False):
         ("IPython", lambda mod: mod.__version__),
         ("sphinx", lambda mod: mod.__version__),
         ("patsy", lambda mod: mod.__version__),
-        ("scikits.timeseries", lambda mod: mod.__version__),
         ("dateutil", lambda mod: mod.__version__),
         ("pytz", lambda mod: mod.VERSION),
         ("bottleneck", lambda mod: mod.__version__),


Test patch:
diff --git a/pandas/stats/tests/test_moments.py b/pandas/stats/tests/test_moments.py
--- a/pandas/stats/tests/test_moments.py
+++ b/pandas/stats/tests/test_moments.py
@@ -65,47 +65,40 @@ def test_rolling_mean(self):
         self._check_moment_func(mom.rolling_mean, np.mean)
 
     def test_cmov_mean(self):
+        # GH 8238
         tm._skip_if_no_scipy()
-        try:
-            from scikits.timeseries.lib import cmov_mean
-        except ImportError:
-            raise nose.SkipTest("no scikits.timeseries")
 
-        vals = np.random.randn(10)
-        xp = cmov_mean(vals, 5)
+        vals = np.array([6.95, 15.21, 4.72, 9.12, 13.81, 13.49, 
+                         16.68, 9.48, 10.63, 14.48])
+        xp = np.array([np.nan, np.nan, 9.962, 11.27 , 11.564, 12.516,
+                       12.818,  12.952, np.nan, np.nan])
 
         rs = mom.rolling_mean(vals, 5, center=True)
-        assert_almost_equal(xp.compressed(), rs[2:-2])
-        assert_almost_equal(xp.mask, np.isnan(rs))
+        assert_almost_equal(xp, rs)
 
         xp = Series(rs)
         rs = mom.rolling_mean(Series(vals), 5, center=True)
         assert_series_equal(xp, rs)
 
     def test_cmov_window(self):
+        # GH 8238
         tm._skip_if_no_scipy()
-        try:
-            from scikits.timeseries.lib import cmov_window
-        except ImportError:
-            raise nose.SkipTest("no scikits.timeseries")
 
-        vals = np.random.randn(10)
-        xp = cmov_window(vals, 5, 'boxcar')
+        vals = np.array([6.95, 15.21, 4.72, 9.12, 13.81, 
+                         13.49, 16.68, 9.48, 10.63, 14.48])
+        xp = np.array([np.nan, np.nan, 9.962, 11.27 , 11.564, 12.516,
+                       12.818,  12.952, np.nan, np.nan])
 
         rs = mom.rolling_window(vals, 5, 'boxcar', center=True)
-        assert_almost_equal(xp.compressed(), rs[2:-2])
-        assert_almost_equal(xp.mask, np.isnan(rs))
+        assert_almost_equal(xp, rs)
 
         xp = Series(rs)
         rs = mom.rolling_window(Series(vals), 5, 'boxcar', center=True)
         assert_series_equal(xp, rs)
 
     def test_cmov_window_corner(self):
+        # GH 8238
         tm._skip_if_no_scipy()
-        try:
-            from scikits.timeseries.lib import cmov_window
-        except ImportError:
-            raise nose.SkipTest("no scikits.timeseries")
 
         # all nan
         vals = np.empty(10, dtype=float)
@@ -125,24 +118,37 @@ def test_cmov_window_corner(self):
         self.assertEqual(len(rs), 5)
 
     def test_cmov_window_frame(self):
+        # Gh 8238
         tm._skip_if_no_scipy()
-        try:
-            from scikits.timeseries.lib import cmov_window
-        except ImportError:
-            raise nose.SkipTest("no scikits.timeseries")
+
+        vals = np.array([[ 12.18,   3.64],
+                         [ 10.18,   9.16],
+                         [ 13.24,  14.61],
+                         [  4.51,   8.11],
+                         [  6.15,  11.44],
+                         [  9.14,   6.21],
+                         [ 11.31,  10.67],
+                         [  2.94,   6.51],
+                         [  9.42,   8.39],
+                         [ 12.44,   7.34 ]])
+
+        xp = np.array([[ np.nan,  np.nan],
+                       [ np.nan,  np.nan],
+                       [  9.252,   9.392],
+                       [  8.644,   9.906],
+                       [  8.87 ,  10.208],
+                       [  6.81 ,   8.588],
+                       [  7.792,   8.644],
+                       [  9.05 ,   7.824],
+                       [ np.nan,  np.nan],
+                       [ np.nan,  np.nan]])
 
         # DataFrame
-        vals = np.random.randn(10, 2)
-        xp = cmov_window(vals, 5, 'boxcar')
         rs = mom.rolling_window(DataFrame(vals), 5, 'boxcar', center=True)
         assert_frame_equal(DataFrame(xp), rs)
 
     def test_cmov_window_na_min_periods(self):
         tm._skip_if_no_scipy()
-        try:
-            from scikits.timeseries.lib import cmov_window
-        except ImportError:
-            raise nose.SkipTest("no scikits.timeseries")
 
         # min_periods
         vals = Series(np.random.randn(10))
@@ -155,39 +161,136 @@ def test_cmov_window_na_min_periods(self):
         assert_series_equal(xp, rs)
 
     def test_cmov_window_regular(self):
+        # GH 8238
         tm._skip_if_no_scipy()
-        try:
-            from scikits.timeseries.lib import cmov_window
-        except ImportError:
-            raise nose.SkipTest("no scikits.timeseries")
 
         win_types = ['triang', 'blackman', 'hamming', 'bartlett', 'bohman',
                      'blackmanharris', 'nuttall', 'barthann']
+
+        vals = np.array([6.95, 15.21, 4.72, 9.12, 13.81,
+                         13.49, 16.68, 9.48, 10.63, 14.48])
+        xps = {
+            'hamming': [np.nan, np.nan, 8.71384, 9.56348, 12.38009, 
+                        14.03687, 13.8567, 11.81473, np.nan, np.nan], 
+            'triang': [np.nan, np.nan, 9.28667, 10.34667, 12.00556, 
+                       13.33889, 13.38, 12.33667, np.nan, np.nan], 
+            'barthann': [np.nan, np.nan, 8.4425, 9.1925, 12.5575, 
+                         14.3675, 14.0825, 11.5675, np.nan, np.nan], 
+            'bohman': [np.nan, np.nan, 7.61599, 9.1764, 12.83559, 
+                       14.17267, 14.65923, 11.10401, np.nan, np.nan], 
+            'blackmanharris': [np.nan, np.nan, 6.97691, 9.16438, 13.05052, 
+                               14.02156, 15.10512, 10.74574, np.nan, np.nan],
+            'nuttall': [np.nan, np.nan, 7.04618, 9.16786, 13.02671, 
+                        14.03559, 15.05657, 10.78514, np.nan, np.nan], 
+            'blackman': [np.nan, np.nan, 7.73345, 9.17869, 12.79607, 
+                         14.20036, 14.57726, 11.16988, np.nan, np.nan], 
+            'bartlett': [np.nan, np.nan, 8.4425, 9.1925, 12.5575, 
+                         14.3675, 14.0825, 11.5675, np.nan, np.nan]}
+
         for wt in win_types:
-            vals = np.random.randn(10)
-            xp = cmov_window(vals, 5, wt)
+            xp = Series(xps[wt])
+            rs = mom.rolling_window(Series(vals), 5, wt, center=True)
+            assert_series_equal(xp, rs)
+
+    def test_cmov_window_regular_linear_range(self):
+        # GH 8238
+        tm._skip_if_no_scipy()
 
+        win_types = ['triang', 'blackman', 'hamming', 'bartlett', 'bohman',
+                     'blackmanharris', 'nuttall', 'barthann']
+
+        vals = np.array(range(10), dtype=np.float)
+        xp = vals.copy()
+        xp[:2] = np.nan
+        xp[-2:] = np.nan
+        xp = Series(xp)
+
+        for wt in win_types:
             rs = mom.rolling_window(Series(vals), 5, wt, center=True)
-            assert_series_equal(Series(xp), rs)
+            assert_series_equal(xp, rs)
+
+    def test_cmov_window_regular_missing_data(self):
+        # GH 8238
+        tm._skip_if_no_scipy()
+
+        win_types = ['triang', 'blackman', 'hamming', 'bartlett', 'bohman',
+                     'blackmanharris', 'nuttall', 'barthann']
+
+        vals = np.array([6.95, 15.21, 4.72, 9.12, 13.81, 
+                         13.49, 16.68, np.nan, 10.63, 14.48])
+        xps = {
+            'bartlett': [np.nan, np.nan, 9.70333, 10.5225, 8.4425, 
+                         9.1925, 12.5575, 14.3675, 15.61667, 13.655], 
+            'blackman': [np.nan, np.nan, 9.04582, 11.41536, 7.73345, 
+                         9.17869, 12.79607, 14.20036, 15.8706, 13.655], 
+            'barthann': [np.nan, np.nan, 9.70333, 10.5225, 8.4425, 
+                         9.1925, 12.5575, 14.3675, 15.61667, 13.655], 
+            'bohman': [np.nan, np.nan, 8.9444, 11.56327, 7.61599, 
+                       9.1764, 12.83559, 14.17267, 15.90976, 13.655], 
+            'hamming': [np.nan, np.nan, 9.59321, 10.29694, 8.71384, 
+                        9.56348, 12.38009, 14.20565, 15.24694, 13.69758], 
+            'nuttall': [np.nan, np.nan, 8.47693, 12.2821, 7.04618, 
+                        9.16786, 13.02671, 14.03673, 16.08759, 13.65553], 
+            'triang': [np.nan, np.nan, 9.33167, 9.76125, 9.28667, 
+                       10.34667, 12.00556, 13.82125, 14.49429, 13.765], 
+            'blackmanharris': [np.nan, np.nan, 8.42526, 12.36824, 6.97691,
+                               9.16438, 13.05052, 14.02175, 16.1098, 
+                               13.65509]
+            }
+
+        for wt in win_types:
+            xp = Series(xps[wt])
+            rs = mom.rolling_window(Series(vals), 5, wt, min_periods=3)
+            assert_series_equal(xp, rs)
 
     def test_cmov_window_special(self):
+        # GH 8238
         tm._skip_if_no_scipy()
-        try:
-            from scikits.timeseries.lib import cmov_window
-        except ImportError:
-            raise nose.SkipTest("no scikits.timeseries")
 
         win_types = ['kaiser', 'gaussian', 'general_gaussian', 'slepian']
         kwds = [{'beta': 1.}, {'std': 1.}, {'power': 2., 'width': 2.},
                 {'width': 0.5}]
 
+        vals = np.array([6.95, 15.21, 4.72, 9.12, 13.81,
+                         13.49, 16.68, 9.48, 10.63, 14.48])
+
+        xps = {
+            'gaussian': [np.nan, np.nan, 8.97297, 9.76077, 12.24763, 
+                         13.89053, 13.65671, 12.01002, np.nan, np.nan], 
+            'general_gaussian': [np.nan, np.nan, 9.85011, 10.71589, 
+                                 11.73161, 13.08516, 12.95111, 12.74577, 
+                                 np.nan, np.nan], 
+            'slepian': [np.nan, np.nan, 9.81073, 10.89359, 11.70284, 
+                        12.88331, 12.96079, 12.77008, np.nan, np.nan], 
+            'kaiser': [np.nan, np.nan, 9.86851, 11.02969, 11.65161, 
+                       12.75129, 12.90702, 12.83757, np.nan, np.nan]
+        }
+
         for wt, k in zip(win_types, kwds):
-            vals = np.random.randn(10)
-            xp = cmov_window(vals, 5, (wt,) + tuple(k.values()))
+            xp = Series(xps[wt])
 
             rs = mom.rolling_window(Series(vals), 5, wt, center=True,
                                     **k)
-            assert_series_equal(Series(xp), rs)
+            assert_series_equal(xp, rs)
+
+    def test_cmov_window_special_linear_range(self):
+        # GH 8238
+        tm._skip_if_no_scipy()
+
+        win_types = ['kaiser', 'gaussian', 'general_gaussian', 'slepian']
+        kwds = [{'beta': 1.}, {'std': 1.}, {'power': 2., 'width': 2.},
+                {'width': 0.5}]
+
+        vals = np.array(range(10), dtype=np.float)
+        xp = vals.copy()
+        xp[:2] = np.nan
+        xp[-2:] = np.nan
+        xp = Series(xp)
+
+        for wt, k in zip(win_types, kwds):
+            rs = mom.rolling_window(Series(vals), 5, wt, center=True,
+                                    **k)
+            assert_series_equal(xp, rs)
 
     def test_rolling_median(self):
         self._check_moment_func(mom.rolling_median, np.median)