Repository: pandas-dev/pandas

Issue:
Unexpected behavior for rolling moments when center=True and min_periods < window
I noticed while using rolling moment functions with `center=True` and with `min_periods` < `window` that the values at the end of the series were showing up as `NaN` when they should have be finite floating point values. The reason for this behavior is that `_rolling_moment()` concats extra `NaN` values to the end of the input series when `center=True` to allow the `roll_generic()` function to continue on past the end of the original series and compute the values that will "come within view" when the result is centered. However, adding those `NaN` values into the window can bork the calculation and change a result that should be finite into a `NaN`.

This example illustrates the behavior:

```
n = 12
s = Series(np.random.randn(n))
s.plot(color='b')
win=7
minp = 5
pd.rolling_mean(s, win, min_periods=minp, center=False).plot(color='g')
pd.rolling_mean(s, win, min_periods=minp, center=True).plot(color='r')
ticks = plt.xticks(np.arange(0, n, 1.0))
```

![image](https://cloud.githubusercontent.com/assets/7545449/4264568/d93c7a9e-3c2c-11e4-843e-ac7914935ff7.png)

I was able to mitigate this behavior, for one calculation at least, by modifying the cython function to essentially strip the trailing `NaN` values for the calculations made towards the end of the series. This requires passing in the `offset` calculated in `_rolling_moment()`. 

Fixing this is going to touch a lot of functions that depend on `_rolling_moment()`. Before I go much further with this, I thought I'd raise the issue here and see if anyone else has insight or better ideas about how to fix this.



Discussion:
@seth-p I believe you added the additional `NaN` values feature to `_rolling_moment()`. Would you mind taking a look at this?

@stahlous always `pd.show_versions()`

Will try to take a look later today or tomorrow.

> On Sep 14, 2014, at 12:55 PM, stahlous notifications@github.com wrote:
> 
> @seth-p I believe you added the additional NaN values feature to _rolling_moment(). Would you mind taking a look at this?
> 
> —
> Reply to this email directly or view it on GitHub.

Ack. I think I might be misleading you by convoluting 0.14.1 with the dev version. However, the problem does stand for `rolling_apply()` when the aggregating function will (likely) be naive about the presence of trailing `NaN` values. 

My apologies. I first noticed this with `rolling_apply()` and then I started working with the latest stable and thought I had discovered that the problem was more general.

I'll re-state the issue this evening when I am able to get in front of my dev box again.

I think this has been addressed in master (for 0.15.0) in https://github.com/pydata/pandas/pull/7934. Let me know if you still see a problem in master.

I will try to check tonight/tomorrow if still need to fix rolling_apply.

Sorry for the confusion. I did confirm that `rolling_mean()` does work as expected in the dev version. I should've stuck to what I knew rather than digging further and confusing things. That said, `rolling_apply()` does suffer from the problem I described:

```
s = pd.Series(np.random.randn(12))
win = 7
minp = 5
print(pd.rolling_apply(s, win, np.mean, min_periods=minp, center=False))
print(pd.rolling_apply(s, win, np.mean, min_periods=minp, center=True))
```

Outputs:

```
0          NaN
1          NaN
2          NaN
3          NaN
4     0.321198
5     0.593657
6     0.566094
7     0.481168
8     0.404456
9     0.212539
10    0.229766
11    0.086256
dtype: float64

0          NaN
1     0.321198
2     0.593657
3     0.566094
4     0.481168
5     0.404456
6     0.212539
7     0.229766
8     0.086256
9          NaN
10         NaN
11         NaN
dtype: float64
```

---

```
pd.show_versions()
```

Outputs:

```
INSTALLED VERSIONS
------------------
commit: None
python: 3.4.1.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-29-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.14.1-430-g35a9527
nose: None
Cython: 0.21
numpy: 1.9.0
scipy: 0.14.0
statsmodels: None
IPython: 2.2.0
sphinx: None
patsy: None
scikits.timeseries: None
dateutil: 2.1
pytz: 2014.7
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.4.0
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
rpy2: None
sqlalchemy: None
pymysql: None
psycopg2: None
```

I was able to hack together a patch for this in `roll_generic()`, but it requires passing in `offset` from `_rolling_moment()`, which would break its API for all the other functions using it.

```
n = len(input)
for i in range(int_min(win, n)):
    if counts[i] >= minp:
        output[i] = func(input[int_max(i - win + 1, 0) : i + 1], *args,
                         **kwargs)
    else:
        output[i] = NaN

for i in range(win, n - offset):
    buf = buf + 1
    bufarr.data = <char*> buf
    if counts[i] >= minp:
        output[i] = func(bufarr, *args, **kwargs)
    else:
        output[i] = NaN

if offset > 0:
    for j, i in enumerate(range(n - offset, n)):
        if counts[i] >= minp:
            output[i] = func(input[int_max(i - win + 1, 0) : i - j], 
                               *args, **kwargs)
        else:
            output[i] = NaN
```

The easier fix would be to have `rolling_apply()` perform its own call to the cython function rather than using `_rolling_moment()` as an intermediary, but that's not very DRY. The alternatives that I can think of would involve changing the `_rolling_moment()` API, which might be more trouble than it's worth.

Ok, I will take a look and fix.

> On Sep 14, 2014, at 7:41 PM, stahlous notifications@github.com wrote:
> 
> Sorry for the confusion. I did confirm that rolling_mean() does work as expected in the dev version. I should've stuck to what I knew rather than digging further and confusing things. That said, rolling_apply() does suffer from the problem I described:
> 
> s = pd.Series(np.random.randn(12))
> win = 7
> minp = 5
> print(pd.rolling_apply(s, win, np.mean, min_periods=minp, center=False))
> print(pd.rolling_apply(s, win, np.mean, min_periods=minp, center=True))
> 
> 0          NaN
> 1          NaN
> 2          NaN
> 3          NaN
> 4     0.321198
> 5     0.593657
> 6     0.566094
> 7     0.481168
> 8     0.404456
> 9     0.212539
> 10    0.229766
> 11    0.086256
> dtype: float64
> 
> 0          NaN
> 1     0.321198
> 2     0.593657
> 3     0.566094
> 4     0.481168
> 5     0.404456
> 6     0.212539
> 7     0.229766
> 8     0.086256
> 9          NaN
> 10         NaN
> 11         NaN
> dtype: float64
> pd.show_versions()
> 
> ## INSTALLED VERSIONS
> 
> commit: None
> python: 3.4.1.final.0
> python-bits: 64
> OS: Linux
> OS-release: 3.13.0-29-generic
> machine: x86_64
> processor: x86_64
> byteorder: little
> LC_ALL: None
> LANG: en_US.UTF-8
> 
> pandas: 0.14.1-430-g35a9527
> nose: None
> Cython: 0.21
> numpy: 1.9.0
> scipy: 0.14.0
> statsmodels: None
> IPython: 2.2.0
> sphinx: None
> patsy: None
> scikits.timeseries: None
> dateutil: 2.1
> pytz: 2014.7
> bottleneck: None
> tables: None
> numexpr: None
> matplotlib: 1.4.0
> openpyxl: None
> xlrd: None
> xlwt: None
> xlsxwriter: None
> lxml: None
> bs4: None
> html5lib: None
> httplib2: None
> apiclient: None
> rpy2: None
> sqlalchemy: None
> pymysql: None
> psycopg2: None
> I was able to hack together a patch for this in roll_generic(), but it requires passing in offset from _rolling_moment(), which would break its API for all the other functions using it.
> 
> n = len(input)
> for i in range(int_min(win, n)):
>     if counts[i] >= minp:
>         output[i] = func(input[int_max(i - win + 1, 0) : i + 1], _args,
>                          *_kwargs)
>     else:
>         output[i] = NaN
> 
> for i in range(win, n - offset):
>     buf = buf + 1
>     bufarr.data = <char*> buf
>     if counts[i] >= minp:
>         output[i] = func(bufarr, _args, *_kwargs)
>     else:
>         output[i] = NaN
> 
> if offset > 0:
>     for j, i in enumerate(range(n - offset, n)):
>         if counts[i] >= minp:
>             output[i] = func(input[int_max(i - win + 1, 0) : i - j], 
>                                _args, *_kwargs)
>         else:
>             output[i] = NaN
> The easier fix would be to have rolling_apply() perform its own call to the cython function rather than using _rolling_moment() as an intermediary, but that's not very DRY. The alternatives that I can think of would involve changing the _rolling_moment() API, which might be more trouble than it's worth.
> 
> —
> Reply to this email directly or view it on GitHub.

OK, I see what the problem is. When `center=True`, all the `rolling_*()` functions pad the end with `NaN` values, so that a window of length `window` centered on the last entry will be "full". The problem is that `np.mean` will return `NaN` if it encounters any `NaN` values in the input. You won't have this problem if you use `np.nanmean` instead. (None of the `rolling_*()` functions besides `rolling_apply` have a problem, because they don't rely on the numpy functions that return `NaN` whenever they encounter a `NaN`.)

The current behavior is asymmetric in the sense that no `NaN` values are pre-pended at the beginning of the series, and so `rolling_apply(..., np.mean, center=True)` does calculate values at the start before there is a full window; and so it should have similar behavior at the end. So I guess my implementation of #7934 to fix #7925 wasn't ideal.

Will try to come up with a different implementation that doesn't have this issue.

I actually have a bit of a philosophical question about this. For simplicity, let's concentrate on the start rather than the edge of the array. Consider:

```
pd.rolling_apply(pd.Series([5., 6., 7., 8., 9.]), 3,
                 lambda v: np.nansum([(i+1)*x for i, x in enumerate(v)]),
                 min_periods=1, center=True)
```

It's not obvious to me what we would want the first entry of the result to be. I can see two perfectly plausible possibilities:
(a) `np.nansum([1*5., 2*6.] = 17.`. -- this is the current behavior
(b) `np.nansum([1*NaN, 2*5., 3*6.] = 28.`.

Currently (in master) the start of the array is treated as (a), but end of the array is treated as (b) (with `NaN` padding the end rather than the beginning of the sub-array, of course). Presumably they should be consistent, but to be honest it's not obvious to me which is "right".

Note that it only makes a difference (a) for functions that don't ignore `NaN` (like your example, `np.mean`), and (b) for functions that are sensitive to the absolute position of the arguments, like my example here (or, for example, if one tried to implement a rolling EWMA).

I guess I'm slightly inclined to keep behavior (a) (and make it the behavior at the end as well) for backwards-compatibility reasons, but I'm not sure it's obvious...

CCing @nbdanielson, since he raised https://github.com/pydata/pandas/issues/7925.

I think option (a) is the more robust choice, but adding `NaN`s is the more efficient choice - for most use cases. I think the answer is to treat `rolling_apply()` as a special case and create a solution that doesn't involve padding with `NaN`s. Otherwise all the rolling moment functions would have to be edited to handle the end of the series differently.

Yes, implementation-wise I am working on a slightly different implementation for `rolling_apply()`. But just want to make sure no one objects too vehemently to behavior (a).


Resolving patch:
diff --git a/pandas/stats/moments.py b/pandas/stats/moments.py
--- a/pandas/stats/moments.py
+++ b/pandas/stats/moments.py
@@ -738,11 +738,12 @@ def rolling_apply(arg, window, func, min_periods=None, freq=None,
     frequency by resampling the data. This is done with the default parameters
     of :meth:`~pandas.Series.resample` (i.e. using the `mean`).
     """
+    offset = int((window - 1) / 2.) if center else 0
     def call_cython(arg, window, minp, args, kwargs):
         minp = _use_window(minp, window)
-        return algos.roll_generic(arg, window, minp, func, args, kwargs)
+        return algos.roll_generic(arg, window, minp, offset, func, args, kwargs)
     return _rolling_moment(arg, window, call_cython, min_periods, freq=freq,
-                           center=center, args=args, kwargs=kwargs)
+                           center=False, args=args, kwargs=kwargs)
 
 
 def rolling_window(arg, window=None, win_type=None, min_periods=None,


Test patch:
diff --git a/pandas/stats/tests/test_moments.py b/pandas/stats/tests/test_moments.py
--- a/pandas/stats/tests/test_moments.py
+++ b/pandas/stats/tests/test_moments.py
@@ -5,6 +5,7 @@
 from datetime import datetime
 from numpy.random import randn
 import numpy as np
+from distutils.version import LooseVersion
 
 from pandas import Series, DataFrame, Panel, bdate_range, isnull, notnull
 from pandas.util.testing import (
@@ -877,6 +878,45 @@ def _ewma(s, com, min_periods, adjust, ignore_na):
                         var_debiasing_factors=lambda x: _variance_debiasing_factors(x, com=com, adjust=adjust, ignore_na=ignore_na))
 
     def test_expanding_consistency(self):
+        base_functions = [
+            (mom.expanding_count, lambda v: Series(v).count(), None),
+            (mom.expanding_max, lambda v: Series(v).max(), None),
+            (mom.expanding_min, lambda v: Series(v).min(), None),
+            (mom.expanding_sum, lambda v: Series(v).sum(), None),
+            (mom.expanding_mean, lambda v: Series(v).mean(), None),
+            (mom.expanding_std, lambda v: Series(v).std(), 1),
+            (mom.expanding_cov, lambda v: Series(v).cov(Series(v)), None),
+            (mom.expanding_corr, lambda v: Series(v).corr(Series(v)), None),
+            (mom.expanding_var, lambda v: Series(v).var(), 1),
+            #(mom.expanding_skew, lambda v: Series(v).skew(), 3), # restore once GH 8086 is fixed
+            #(mom.expanding_kurt, lambda v: Series(v).kurt(), 4), # restore once GH 8086 is fixed
+            #(lambda x, min_periods: mom.expanding_quantile(x, 0.3, min_periods=min_periods),
+            # lambda v: Series(v).quantile(0.3), None), # restore once GH 8084 is fixed
+            (mom.expanding_median, lambda v: Series(v).median(), None),
+            (mom.expanding_max, np.nanmax, 1),
+            (mom.expanding_min, np.nanmin, 1),
+            (mom.expanding_sum, np.nansum, 1),
+            ]
+        if np.__version__ >= LooseVersion('1.8.0'):
+            base_functions += [
+                (mom.expanding_mean, np.nanmean, 1),
+                (mom.expanding_std, lambda v: np.nanstd(v, ddof=1), 1),
+                (mom.expanding_var, lambda v: np.nanvar(v, ddof=1), 1),
+            ]
+            if np.__version__ >= LooseVersion('1.9.0'):
+                base_functions += [
+                    (mom.expanding_median, np.nanmedian, 1),
+                ]
+        no_nan_functions = [
+            (mom.expanding_max, np.max, None),
+            (mom.expanding_min, np.min, None),
+            (mom.expanding_sum, np.sum, None),
+            (mom.expanding_mean, np.mean, None),
+            (mom.expanding_std, lambda v: np.std(v, ddof=1), 1),
+            (mom.expanding_var, lambda v: np.var(v, ddof=1), 1),
+            (mom.expanding_median, np.median, None),
+            ]
+
         for min_periods in [0, 1, 2, 3, 4]:
 
             # test consistency between different expanding_* moments
@@ -895,25 +935,15 @@ def test_expanding_consistency(self):
                 var_debiasing_factors=lambda x: mom.expanding_count(x) / (mom.expanding_count(x) - 1.).replace(0., np.nan)
                 )
 
-            # test consistency between expanding_xyz() and expanding_apply of Series/DataFrame.xyz()
+            # test consistency between expanding_xyz() and either (a) expanding_apply of Series.xyz(),
+            #                                                  or (b) expanding_apply of np.nanxyz()
             for x in self._test_data():
                 assert_equal = assert_series_equal if isinstance(x, Series) else assert_frame_equal
-                for (expanding_f, f, require_min_periods) in [
-                    (mom.expanding_count, lambda v: Series(v).count(), None),
-                    (mom.expanding_max, lambda v: Series(v).max(), None),
-                    (mom.expanding_min, lambda v: Series(v).min(), None),
-                    (mom.expanding_sum, lambda v: Series(v).sum(), None),
-                    (mom.expanding_mean, lambda v: Series(v).mean(), None),
-                    (mom.expanding_std, lambda v: Series(v).std(), 1),
-                    (mom.expanding_cov, lambda v: Series(v).cov(Series(v)), None),
-                    (mom.expanding_corr, lambda v: Series(v).corr(Series(v)), None),
-                    (mom.expanding_var, lambda v: Series(v).var(), 1),
-                    #(mom.expanding_skew, lambda v: Series(v).skew(), 3), # restore once GH 8086 is fixed
-                    #(mom.expanding_kurt, lambda v: Series(v).kurt(), 4), # restore once GH 8086 is fixed
-                    #(lambda x, min_periods: mom.expanding_quantile(x, 0.3, min_periods=min_periods),
-                    # lambda v: Series(v).quantile(0.3), None), # restore once GH 8084 is fixed
-                    (mom.expanding_median, lambda v: Series(v).median(), None),
-                    ]:
+                functions = base_functions
+                # GH 8269
+                if x.notnull().all().all():
+                    functions = base_functions + no_nan_functions
+                for (expanding_f, f, require_min_periods) in functions:
                     if require_min_periods and (min_periods is not None) and (min_periods < require_min_periods):
                         continue
 
@@ -938,7 +968,46 @@ def test_expanding_consistency(self):
                         assert_panel_equal(expanding_f_result, expected)
 
     def test_rolling_consistency(self):
-        for window in [1, 3, 10, 20]:
+        base_functions = [
+            (mom.rolling_count, lambda v: Series(v).count(), None),
+            (mom.rolling_max, lambda v: Series(v).max(), None),
+            (mom.rolling_min, lambda v: Series(v).min(), None),
+            (mom.rolling_sum, lambda v: Series(v).sum(), None),
+            (mom.rolling_mean, lambda v: Series(v).mean(), None),
+            (mom.rolling_std, lambda v: Series(v).std(), 1),
+            (mom.rolling_cov, lambda v: Series(v).cov(Series(v)), None),
+            (mom.rolling_corr, lambda v: Series(v).corr(Series(v)), None),
+            (mom.rolling_var, lambda v: Series(v).var(), 1),
+            #(mom.rolling_skew, lambda v: Series(v).skew(), 3), # restore once GH 8086 is fixed
+            # (mom.rolling_kurt, lambda v: Series(v).kurt(), 4), # restore once GH 8086 is fixed
+            #(lambda x, window, min_periods, center: mom.rolling_quantile(x, window, 0.3, min_periods=min_periods, center=center),
+            # lambda v: Series(v).quantile(0.3), None), # restore once GH 8084 is fixed
+            (mom.rolling_median, lambda v: Series(v).median(), None),
+            (mom.rolling_max, np.nanmax, 1),
+            (mom.rolling_min, np.nanmin, 1),
+            (mom.rolling_sum, np.nansum, 1),
+            ]
+        if np.__version__ >= LooseVersion('1.8.0'):
+            base_functions += [
+                (mom.rolling_mean, np.nanmean, 1),
+                (mom.rolling_std, lambda v: np.nanstd(v, ddof=1), 1),
+                (mom.rolling_var, lambda v: np.nanvar(v, ddof=1), 1),
+            ]
+            if np.__version__ >= LooseVersion('1.9.0'):
+                base_functions += [
+                    (mom.rolling_median, np.nanmedian, 1),
+                ]
+        no_nan_functions = [
+            (mom.rolling_max, np.max, None),
+            (mom.rolling_min, np.min, None),
+            (mom.rolling_sum, np.sum, None),
+            (mom.rolling_mean, np.mean, None),
+            (mom.rolling_std, lambda v: np.std(v, ddof=1), 1),
+            (mom.rolling_var, lambda v: np.var(v, ddof=1), 1),
+            (mom.rolling_median, np.median, None),
+            ]
+
+        for window in [1, 2, 3, 10, 20]:
             for min_periods in set([0, 1, 2, 3, 4, window]):
                 if min_periods and (min_periods > window):
                     continue
@@ -962,25 +1031,15 @@ def test_rolling_consistency(self):
                                                         (mom.rolling_count(x, window=window, center=center) - 1.).replace(0., np.nan)),
                         )
 
-                    # test consistency between rolling_xyz and rolling_apply of Series/DataFrame.xyz
+                    # test consistency between rolling_xyz() and either (a) rolling_apply of Series.xyz(),
+                    #                                                or (b) rolling_apply of np.nanxyz()
                     for x in self._test_data():
                         assert_equal = assert_series_equal if isinstance(x, Series) else assert_frame_equal
-                        for (rolling_f, f, require_min_periods) in [
-                            (mom.rolling_count, lambda v: Series(v).count(), None),
-                            (mom.rolling_max, lambda v: Series(v).max(), None),
-                            (mom.rolling_min, lambda v: Series(v).min(), None),
-                            (mom.rolling_sum, lambda v: Series(v).sum(), None),
-                            (mom.rolling_mean, lambda v: Series(v).mean(), None),
-                            (mom.rolling_std, lambda v: Series(v).std(), 1),
-                            (mom.rolling_cov, lambda v: Series(v).cov(Series(v)), None),
-                            (mom.rolling_corr, lambda v: Series(v).corr(Series(v)), None),
-                            (mom.rolling_var, lambda v: Series(v).var(), 1),
-                            #(mom.rolling_skew, lambda v: Series(v).skew(), 3), # restore once GH 8086 is fixed
-                            # (mom.rolling_kurt, lambda v: Series(v).kurt(), 4), # restore once GH 8086 is fixed
-                            #(lambda x, window, min_periods, center: mom.rolling_quantile(x, window, 0.3, min_periods=min_periods, center=center),
-                            # lambda v: Series(v).quantile(0.3), None), # restore once GH 8084 is fixed
-                            (mom.rolling_median, lambda v: Series(v).median(), None),
-                            ]:
+                        functions = base_functions
+                        # GH 8269
+                        if x.notnull().all().all():
+                            functions = base_functions + no_nan_functions
+                        for (rolling_f, f, require_min_periods) in functions:
                             if require_min_periods and (min_periods is not None) and (min_periods < require_min_periods):
                                 continue