Repository: pandas-dev/pandas
Pull request: [BUG]: Rolling.sum() calculated wrong values when axis is one and dtypes are mixed
- [x] closes #20649
- [x] closes #35596
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

In case of ``axis=1`` and mixed dtypes, the ``_apply_blockwise`` did not calculate the sum for a complete row. Through converting the obj and consolidating the blocks we can avoid this.

Problem statement:
pandas.DataFrame.rolling produces incorrect value if integer and float objects are mixed
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd

# Generate square dataframe of ones
df = pd.DataFrame(1, index=[1,2,3,4,5], columns=['a','b','c','d','e'])

# Make last column a series of type "float" instead of type "int"
df['e'] = 1.0

# Compute rolling sum across rows
print(df.rolling(window=2, min_periods=1, axis=1).sum())

# Output. Note that the last column is incorrect.
# -----------------------------------------------
#      a    b    c    d    e
# 1  1.0  2.0  2.0  2.0  1.0
# 2  1.0  2.0  2.0  2.0  1.0
# 3  1.0  2.0  2.0  2.0  1.0
# 4  1.0  2.0  2.0  2.0  1.0
# 5  1.0  2.0  2.0  2.0  1.0
```
#### Problem description

In the above example, the last column of `df` was coerced into a `float` Series. Calculating a rolling sum of `df` produces an incorrect result for that last column. Incidentally, the correct result is produced by coercing the entire dataframe to a `float` object before computing the rolling sum.

#### Expected Output

```python
# Compute same rolling sum across rows, except convert all values to type "float" first
print(df.astype('float').rolling(window=2, min_periods=1, axis=1).sum())

# Output:
# -------
#      a    b    c    d    e
# 1  1.0  2.0  2.0  2.0  2.0
# 2  1.0  2.0  2.0  2.0  2.0
# 3  1.0  2.0  2.0  2.0  2.0
# 4  1.0  2.0  2.0  2.0  2.0
# 5  1.0  2.0  2.0  2.0  2.0
```

#### Output of ``pd.show_versions()``

<details>


INSTALLED VERSIONS
------------------
commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.22.0
pytest: 3.3.2
pip: 9.0.1
setuptools: 38.4.0
Cython: 0.27.3
numpy: 1.14.0
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: 1.6.6
patsy: 0.5.0
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.4
feather: None
matplotlib: 2.1.2
openpyxl: 2.4.10
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml: 4.1.1
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.1
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>


How to stop a rolling window at nan values and continue after it? 
Posted the same question on stackoverflow. A user there said I should open a issue here on the github page, since it is a bug. 

 I have the following dataframe:

```
     df = pd.DataFrame([[0, 1, 2, 4, np.nan, np.nan, np.nan],
                   [0, 1, 2 ,np.nan, np.nan, np.nan,np.nan],
                   [0, 2, 2 ,np.nan, 2, np.nan,1]])
```
 
With output:

```
       0  1  2    3    4   5   6
    0  0  1  2  4.0  NaN NaN NaN
    1  0  1  2  NaN  NaN NaN NaN
    2  0  2  2  NaN  2.0 NaN 1.0
```

with dtypes:
`
        df.dtypes`

```
    0      int64
    1      int64
    2      int64
    3    float64
    4    float64
    5    float64
    6    float64
    dtype: object
```

Then the underneath rolling summation is applied:

    df.rolling(window = 7, min_periods =1, axis = 'columns').sum()

And the output is as follows:

```
         0    1    2    3    4    5    6
    0  0.0  1.0  3.0  4.0  4.0  4.0  4.0
    1  0.0  1.0  3.0  NaN  NaN  NaN  NaN
    2  0.0  2.0  4.0  NaN  2.0  2.0  3.0
```

I notice that the rolling window stops and starts again whenever the `dtype` of the next column is different. 

I however have a dataframe whereby all columns are of the same `object` type. 
`
`    df = df.astype('object')``

which has output:

```
         0    1    2    3    4    5    6
    0  0.0  1.0  3.0  7.0  7.0  7.0  7.0
    1  0.0  1.0  3.0  3.0  3.0  3.0  3.0
    2  0.0  2.0  4.0  4.0  6.0  6.0  7.0
```

My desired output however, stops and starts again after a `nan` value appears. This would look like:
```

         0    1    2    3    4    5    6
    0  0.0  1.0  3.0  7.0  NaN  NaN  NaN
    1  0.0  1.0  3.0  NaN  NaN  NaN  NaN
    2  0.0  2.0  4.0  NaN  2.0  NaN  3.0
```

I figured there must be a way that NaN values are not considered but also not filled in with values obtained from the rolling window. 

Anything would help!

Hints from issue discussion:
I'm trying to understand your desired output. You say the rolling sum stops and starts again after a nan appears which, given the last value in the row is 3.0 and not 1.0, appears to mean the first observed nan? 
cc @mroeschke OP says this is a bug; do you agree?  if so pls add label
Yeah this looks like a bug. `axis='columns'` doesn't have much testing and I would expect something similar to 

```
In [10]: df.T.rolling(window=7, min_periods=1).sum().T
Out[10]:
     0    1    2    3    4    5    6
0  0.0  1.0  3.0  7.0  7.0  7.0  7.0
1  0.0  1.0  3.0  3.0  3.0  3.0  3.0
2  0.0  2.0  4.0  4.0  6.0  6.0  7.0
```

Commit message: Fix bug with mixed dtypes and axis equals 1

Patch to review:
diff --git a/doc/source/whatsnew/v1.2.0.rst b/doc/source/whatsnew/v1.2.0.rst
index 3992e697db7e4..36f1e32e0a251 100644
--- a/doc/source/whatsnew/v1.2.0.rst
+++ b/doc/source/whatsnew/v1.2.0.rst
@@ -336,7 +336,7 @@ Groupby/resample/rolling
 - Bug in :meth:`DataFrameGroupby.tshift` failing to raise ``ValueError`` when a frequency cannot be inferred for the index of a group (:issue:`35937`)
 - Bug in :meth:`DataFrame.groupby` does not always maintain column index name for ``any``, ``all``, ``bfill``, ``ffill``, ``shift`` (:issue:`29764`)
 - Bug in :meth:`DataFrameGroupBy.apply` raising error with ``np.nan`` group(s) when ``dropna=False`` (:issue:`35889`)
--
+- Bug in :meth:`Rolling.sum()` returned wrong values when dtypes where mixed between float and integer and axis was equal to one (:issue:`20649`)
 
 Reshaping
 ^^^^^^^^^
diff --git a/pandas/core/window/rolling.py b/pandas/core/window/rolling.py
index 21a7164411fb7..13b448c3b07b3 100644
--- a/pandas/core/window/rolling.py
+++ b/pandas/core/window/rolling.py
@@ -25,6 +25,7 @@
 from pandas._typing import ArrayLike, Axis, FrameOrSeries, FrameOrSeriesUnion
 from pandas.compat._optional import import_optional_dependency
 from pandas.compat.numpy import function as nv
+from pandas.core.dtypes.cast import infer_dtype_from
 from pandas.util._decorators import Appender, Substitution, cache_readonly, doc
 
 from pandas.core.dtypes.common import (
@@ -490,6 +491,9 @@ def _apply_blockwise(
             return self._apply_series(homogeneous_func)
 
         obj = self._create_data(self._selected_obj)
+        if self.axis == 1:
+            obj = obj.astype(infer_dtype_from(obj.values)[0], copy=False)
+            obj._mgr = obj._mgr.consolidate()
         mgr = obj._mgr
 
         def hfunc(bvalues: ArrayLike) -> ArrayLike:
diff --git a/pandas/tests/window/test_rolling.py b/pandas/tests/window/test_rolling.py
index 88afcec0f7bf4..0063913a75b84 100644
--- a/pandas/tests/window/test_rolling.py
+++ b/pandas/tests/window/test_rolling.py
@@ -771,3 +771,18 @@ def test_rolling_numerical_too_large_numbers():
         index=dates,
     )
     tm.assert_series_equal(result, expected)
+
+
+@pytest.mark.parametrize(
+    ("func", "value"),
+    [("sum", 2.0), ("max", 1.0), ("min", 1.0), ("mean", 1.0), ("median", 1.0)],
+)
+def test_rolling_mixed_dtypes_axis_1(func, value):
+    # GH: 20649
+    df = pd.DataFrame(1, index=[1, 2], columns=["a", "b", "c"])
+    df["c"] = 1.0
+    result = getattr(df.rolling(window=2, min_periods=1, axis=1), func)()
+    expected = pd.DataFrame(
+        {"a": [1.0, 1.0], "b": [value, value], "c": [value, value]}, index=[1, 2]
+    )
+    tm.assert_frame_equal(result, expected)


Code review comments:
diff_hunk: @@ -490,6 +491,9 @@ def _apply_blockwise(
             return self._apply_series(homogeneous_func)
 
         obj = self._create_data(self._selected_obj)
+        if self.axis == 1:
+            obj = obj.astype(infer_dtype_from(obj.values)[0], copy=False)
path: pandas/core/window/rolling.py
text: @user1:
This logic may be better suited in `_create_data` so other operations also benefit when axis=1

@user1:
Also generally we require all input data to be castable to float (see the `_prep_values` function).

So instead of using `infer_dtype_from` I think you can just try to cast directly to float instead. 

@author:
I found an issue discussing adding support for other dtypes. I wanted to be on the save side with regard to this. But I change it then.

Additionally I was not sure if this should affect functions outside of ``_apply_blockwise``. I moved this into ``_create_data``

@author:
Ähm I got one question about the dtype thing: How can we convert to float32, if float32 is the input and float64 otherwise? Just giving "float" into astype converts it to "float64". Or does the function convert the dtype to "float32" nevertheless?

@user1:
Right eventually we would like to support more input data types eventually but that issue can be addressed later.

We generally coerce all inputs to `float64` in `_prep_values` anyways so the 32 vs 64 bit distinction doesn't matter here. So this should always coerce to float64

@author:
Thanks very much. Done