Repository: pandas-dev/pandas

Issue:
pandas.rolling.apply skip calling function if window contains any NaN
#### Code Sample, a copy-pastable example if possible
```python
import numpy as np
import pandas as pd

# sample data with NaN
df = pd.DataFrame([np.arange(8) + i * 10 for i in range(3)]).T
df[0][3] = np.nan
df[1][2] = np.nan
df[2][6] = np.nan
print(df)

# sample function to apply to window
def my_std(x): 
    print(x)
    return x.std()

# apply my_std to rolling window(5)
print(df.rolling(5).apply(my_std, raw=False)) 
```
#### Problem description

I wonder if this is an intended/unintended behavior. Apparently when a `Rolling` object runs the `apply` method, it skips calling the function completely if data in the window contains any `np.nan`. 

`df` looks like this:
```
     0     1     2
0  0.0  10.0  20.0
1  1.0  11.0  21.0
2  2.0   NaN  22.0
3  NaN  13.0  23.0
4  4.0  14.0  24.0
5  5.0  15.0  25.0
6  6.0  16.0   NaN
7  7.0  17.0  27.0
```
Output of `df.rolling(5).apply(my_std, raw=False)`:
```
    0         1         2
0 NaN       NaN       NaN
1 NaN       NaN       NaN
2 NaN       NaN       NaN
3 NaN       NaN       NaN
4 NaN       NaN  1.581139
5 NaN       NaN  1.581139
6 NaN       NaN       NaN
7 NaN  1.581139       NaN
```

I understand that in older versions, pandas calls numpy primitives to handle rolling windows, which leads to NaNs as numpy function propagates it. But here, the NaNs are not caused by `my_std`, because data in the first column are not even printed, i.e. `my_std` is not even called for the first column as there is no single non-NaN window. 

Confusingly, the `Expanding` object would not do this. `Expanding.apply` actually calls the function with the expanding window despite if there is any `NaN` or not. 

#### Expected Output

expected output of running: `df.rolling(5).apply(my_std, raw=False)`
```
          0         1         2
0       NaN       NaN       NaN
1       NaN       NaN       NaN
2       NaN       NaN       NaN
3       NaN       NaN       NaN
4       NaN       NaN  1.581139
5  1.825742  1.707825  1.581139
6  1.707825  1.290994  1.290994
7  1.290994  1.581139  1.707825
```

currently this is achieved by running: `df.expanding(5).apply(lambda x: x[-5:].std(), raw=False)`
it's confusing that one has to use `expanding(5)` to do the job of `rolling(5)` when `NaN` is present in data. Performance-wise is probably not ideal either... 

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 63 Stepping 0, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None
pandas: 0.23.0
pytest: None
pip: 10.0.1
setuptools: 39.2.0
Cython: None
numpy: 1.14.3
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 6.4.0
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.2.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>



Discussion:
The difference you are seeing is a result of `min_periods` which defaults to the length of the window you've provided in the rolling function. In this case, because you don't have 5 non-NA records within your window the function automatically returns NA.

Note the difference in behavior when passing this explicitly:
```python
>>>  df.rolling(5, min_periods=4).apply(my_std, raw=False)
          0         1         2
0       NaN       NaN       NaN
1       NaN       NaN       NaN
2       NaN       NaN       NaN
3       NaN       NaN  1.290994
4  1.707825  1.825742  1.581139
5  1.825742  1.707825  1.581139
6  1.707825  1.290994  1.290994
7  1.290994  1.581139  1.707825
```

See also the rolling documentation:
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.rolling.html

I think this issue is worth leaving open if only to update the documentation of that keyword argument. There is an example that shows how it functions, but the description of that argument could use some clarification on what `None` actually means
Thank you, WillAyd. You are right. It is the effect of `min_periods` indeed! Really appreciate your reply. Should I be leaving this open for a few more days to wait for the change in the documentation to happen? Or should I make change to the documentation myself? Sorry for the questions, this is the first time I posted a question in the forum.. Thanks
If you are open to making the change yourself contributions like that are welcome and encouraged. Just be sure to check out the pandas contributor guide:

https://pandas.pydata.org/pandas-docs/stable/contributing.html
@jwchi, are you planning on making the changes to the documentation or can I take this up?
Is this issue currently being worked on? If not, I'd like to take it up.
@mattboggess : Go for it!  Doesn't look like anyone is currently working on it.

Resolving patch:
diff --git a/pandas/core/window.py b/pandas/core/window.py
--- a/pandas/core/window.py
+++ b/pandas/core/window.py
@@ -462,7 +462,8 @@ class Window(_Window):
     min_periods : int, default None
         Minimum number of observations in window required to have a value
         (otherwise result is NA). For a window that is specified by an offset,
-        this will default to 1.
+        `min_periods` will default to 1. Otherwise, `min_periods` will default
+        to the size of the window.
     center : boolean, default False
         Set the labels at the center of the window.
     win_type : string, default None


Test patch: