Repository: pandas-dev/pandas
Pull request: BUG: Rolling apply on DataFrame with Datetime index returns NaN
 - [x]  #15305
 - [x] changing the order of 'calling roll_sum' and 'get_window_indexer' in 'roll_generic' could fix #15305 
The bug caused by the return values from get_window_indexer. It changes value of 'win', which is not supposed to be passed to roll_sum.




Problem statement:
Rolling apply on DataFrame with Datetime index returns NaN
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd

df = pd.DataFrame({'A': [0, 1, 2, 3, 4]},
                  index=[pd.Timestamp('20130101 09:00:00'),
                         pd.Timestamp('20130101 09:00:01'),
                         pd.Timestamp('20130101 09:00:02'),
                         pd.Timestamp('20130101 09:00:03'),
                         pd.Timestamp('20130101 09:00:04')])
rolled = df.rolling('3s', 3)

print rolled.sum()
print rolled.apply(sum)
```

outputs
```
                       A
2013-01-01 09:00:00  NaN
2013-01-01 09:00:01  NaN
2013-01-01 09:00:02  3.0
2013-01-01 09:00:03  6.0
2013-01-01 09:00:04  9.0

                      A
2013-01-01 09:00:00 NaN
2013-01-01 09:00:01 NaN
2013-01-01 09:00:02 NaN
2013-01-01 09:00:03 NaN
2013-01-01 09:00:04 NaN
```
#### Problem description
rolled.apply.sum() should give the same output as rolled.apply(sum) but instead only returns nan.
 

#### Expected Output
same as rolled.sum()

#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.12.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-59-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.19.2
nose: None
pip: 9.0.1
setuptools: 20.7.0
Cython: None
numpy: 1.12.0
scipy: 0.18.1
statsmodels: 0.6.1
xarray: None
IPython: 5.1.0
sphinx: None
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.10
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.5.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
boto: 2.38.0
pandas_datareader: None
</details>

Hints from issue discussion:
well ``.apply`` rolling on time-specified windows isn't tested very well :<

The issue is that the counts computation is using the fixed interval one (see a few lines above). Instead it needs to bascially call ``counts = roll_count(...)`` more or les.

see https://github.com/pandas-dev/pandas/blob/master/pandas/window.pyx#L1362

pull-requests are welcome!.

note needs testing with multiple min_periods as well.
The above link is broken.
see [https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/window.pyx#L1412](https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/window.pyx#L1412)

Commit message: fix issue #15305, update _libs\window.pyx and tests\test_window.py

Patch to review:
diff --git a/doc/source/whatsnew/v0.21.0.txt b/doc/source/whatsnew/v0.21.0.txt
index 1931ffff4b217..8fa4bcf28b04e 100644
--- a/doc/source/whatsnew/v0.21.0.txt
+++ b/doc/source/whatsnew/v0.21.0.txt
@@ -296,6 +296,8 @@ Groupby/Resample/Rolling
 - Bug in :func:`infer_freq` causing indices with 2-day gaps during the working week to be wrongly inferred as business daily (:issue:`16624`)
 - Bug in ``.rolling(...).quantile()`` which incorrectly used different defaults than :func:`Series.quantile()` and :func:`DataFrame.quantile()` (:issue:`9413`, :issue:`16211`)
 - Bug in ``groupby.transform()`` that would coerce boolean dtypes back to float (:issue:`16875`)
+- Bug in ``.rolling(...).apply(...)`` where when rolling apply on DataFrame
+  with Datetime index and min_periods set to be greater than 1 (:issue:`15305`)
 
 Sparse
 ^^^^^^
diff --git a/pandas/_libs/window.pyx b/pandas/_libs/window.pyx
index 2450eea5500cd..679ed7222b26f 100644
--- a/pandas/_libs/window.pyx
+++ b/pandas/_libs/window.pyx
@@ -1427,16 +1427,17 @@ def roll_generic(ndarray[float64_t, cast=True] input,
     n = len(input)
     if n == 0:
         return input
-
+    
+    counts = roll_sum(np.concatenate([np.isfinite(input).astype(float),
+                                      np.array([0.] * offset)]),
+                      win, minp, index, closed)[offset:]
+    
     start, end, N, win, minp, is_variable = get_window_indexer(input, win,
                                                                minp, index,
                                                                closed,
                                                                floor=0)
-    output = np.empty(N, dtype=float)
 
-    counts = roll_sum(np.concatenate([np.isfinite(input).astype(float),
-                                      np.array([0.] * offset)]),
-                      win, minp, index, closed)[offset:]
+    output = np.empty(N, dtype=float)
 
     if is_variable:
 
diff --git a/pandas/tests/test_window.py b/pandas/tests/test_window.py
index 5ab33bd6cc5e1..7e31bd4e2645a 100644
--- a/pandas/tests/test_window.py
+++ b/pandas/tests/test_window.py
@@ -423,6 +423,27 @@ def test_constructor_with_timedelta_window(self):
             expected = df.rolling('3D').sum()
             tm.assert_frame_equal(result, expected)
 
+    def test_constructor_with_timedelta_window_and_minperiods(self):
+        # GH 15305
+        n = 10
+        df = pd.DataFrame({'value': np.arange(n)},
+                          index=pd.date_range('2015-12-24',
+                                              periods=n,
+                                              freq="D"))
+        expected_data = np.append([np.NaN, 1.], np.arange(3., 27., 3))
+        for window in [timedelta(days=3), pd.Timedelta(days=3)]:
+            result_roll_sum = df.rolling(window=window, min_periods=2).sum()
+            result_roll_generic = df.rolling(window=window,
+                                             min_periods=2).apply(sum)
+            expected = pd.DataFrame({'value': expected_data},
+                                    index=pd.date_range('2015-12-24',
+                                                        periods=n,
+                                                        freq="D"))
+            tm.assert_frame_equal(result_roll_sum, expected)
+            tm.assert_frame_equal(result_roll_generic, expected)
+            expected = df.rolling('3D', min_periods=2).sum()
+            tm.assert_frame_equal(result_roll_generic, expected)
+
     def test_numpy_compat(self):
         # see gh-12811
         r = rwindow.Rolling(Series([2, 4, 6]), window=2)


Code review comments:
diff_hunk: @@ -423,6 +423,27 @@ def test_constructor_with_timedelta_window(self):
             expected = df.rolling('3D').sum()
             tm.assert_frame_equal(result, expected)
 
+    def test_constructor_with_timedelta_window_and_minperiods(self):
+        # GH 15305
+        n = 10
+        df = pd.DataFrame({'value': np.arange(n)},
+                          index=pd.date_range('2015-12-24',
+                                              periods=n,
+                                              freq="D"))
+        expected_data = np.append([np.NaN, 1.], np.arange(3., 27., 3))
+        for window in [timedelta(days=3), pd.Timedelta(days=3)]:
+            result_roll_sum = df.rolling(window=window, min_periods=2).sum()
+            result_roll_generic = df.rolling(window=window,
+                                             min_periods=2).apply(sum)
+            expected = pd.DataFrame({'value': expected_data},
path: pandas/tests/test_window.py
text: @user1:
you can put the expected_data directly in the frame.

diff_hunk: @@ -423,6 +423,27 @@ def test_constructor_with_timedelta_window(self):
             expected = df.rolling('3D').sum()
             tm.assert_frame_equal(result, expected)
 
+    def test_constructor_with_timedelta_window_and_minperiods(self):
+        # GH 15305
+        n = 10
+        df = pd.DataFrame({'value': np.arange(n)},
+                          index=pd.date_range('2015-12-24',
+                                              periods=n,
+                                              freq="D"))
+        expected_data = np.append([np.NaN, 1.], np.arange(3., 27., 3))
+        for window in [timedelta(days=3), pd.Timedelta(days=3)]:
+            result_roll_sum = df.rolling(window=window, min_periods=2).sum()
+            result_roll_generic = df.rolling(window=window,
+                                             min_periods=2).apply(sum)
+            expected = pd.DataFrame({'value': expected_data},
+                                    index=pd.date_range('2015-12-24',
+                                                        periods=n,
+                                                        freq="D"))
+            tm.assert_frame_equal(result_roll_sum, expected)
+            tm.assert_frame_equal(result_roll_generic, expected)
+            expected = df.rolling('3D', min_periods=2).sum()
path: pandas/tests/test_window.py
text: @user1:
what is this last part testing?

@author:
This tests when window == '3D'. I'd better put it together with [timedelta(days=3), pd.Timedelta(days=3)]