Fix #12373: rolling functions raise ValueError on float32 data
State: closed

- [x] closes #12373
- [x] tests added
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry added
- [x] add test for various other dtypes


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
@BranYang can you rebase / update. 

@jreback Sure, working on this and will update soon.

Still working on this. Lots of tests to handle

@jreback Just added bunch of tests to test window behavior on different dtypes. 

Not very sure is this a good idea because all these tests together takes too long to finish.

What do you think about these added tests? Should we also tests `M8[ns],m8[ns],datetime64[ns, UTC],category` like this or should I just write several new cases?

@BranYang yeah don't need to add for FULL tests (e.g. the moments consistency), these are very time consuming. Good idea about _how_ to do it, but need to inherit from a simpler function (that MomentConsistency) is a monster.

@BranYang if you can update would be great. your fix is good, just need to reduce the testing time (e.g. can add a relatively simple set of tests, and refactor a bit later). This caused slow tests to go from 20m to 40m, not a good thing)

@jreback Yes I am working on write a new set of tests to test rolling results on each dtype. 

thanks @BranYang ok, your fix is fine. lmk about putting in simpler tests (for now), can do a refactor later, but going to need this for 0.18.0 (hopefully going to cut a 0.18.0rc2 at the end of the week).

#12481 is the ultimate way to go, but a little more tricky so going to hold off on that a bit.

@jreback Please review the tests this time.

I have to comment out tests for `M8[ns],m8[ns],datetime64[ns, UTC]` because call count on them will raise a type error:

``` Python
import pandas as pd
import numpy as np
d = pd.Series(np.arange(10), dtype='M8[ns]')
d.rolling(window=2).count() # raise a TypeError
```

Do we need count() work for datetime64/timedelta/category dtypes?

Yeah these should all work, but I see they don't work in 0.17.1, so can prob ignore for now.

so instead of _always_ casting to `float`, check for `com.needs_i8_conversion()` if so, then do this. (we will need some more sophisticated versions of this (as some of these ops need to cast back), but let's at least have count working. (this is sort of easy so you _could_ add this in if you want).

```
In [4]: d
Out[4]: 
0   1970-01-01 00:00:00.000000000
1   1970-01-01 00:00:00.000000001
2   1970-01-01 00:00:00.000000002
3   1970-01-01 00:00:00.000000003
4   1970-01-01 00:00:00.000000004
5   1970-01-01 00:00:00.000000005
6   1970-01-01 00:00:00.000000006
7   1970-01-01 00:00:00.000000007
8   1970-01-01 00:00:00.000000008
9   1970-01-01 00:00:00.000000009
dtype: datetime64[ns]

In [5]: Series(d.view('i8').astype('float64')).rolling(window=2).count()
Out[5]: 
0    1.0
1    2.0
2    2.0
3    2.0
4    2.0
5    2.0
6    2.0
7    2.0
8    2.0
9    2.0
dtype: float64

```

@jreback Not very sure if my change for `count` is the best way to make it work. Waiting for comments.

And - I find that when constructing DataFrame by specifying two dtypes it will raise error:

``` Python
import pandas as pd
import numpy as np
df = pd.DataFrame(np.arange(10).reshape((5, 2)), dtype='M8[ns, UTC]')
# TypeError: data type not understood
df2 = pd.DataFrame(np.arange(10).reshape((5, 2)), dtype='category')
# NotImplementedError: > 1 ndim Categorical are not supported at this time
```

So I just do not test this two cases ( Dataframe \* ('M8[ns, UTC]', 'category')).

revert your change for non numeric dtypes
this is another issue and will require some care

@BranYang we have a plethora of these issues already listed here: https://github.com/pydata/pandas/issues/8659 (IIRC the non-numeric handling is the first). Your tests look good now. Just remove the datetimelike/categorical stuff.

FYI, you can't use a numpy constructor for pandas extension dtypes, but the pandas objects accept these happily.

```
In [4]: Series(np.arange(5),dtype='datetime64[ns, UTC]')
Out[4]: 
0             1970-01-01 00:00:00+00:00
1   1970-01-01 00:00:00.000000001+00:00
2   1970-01-01 00:00:00.000000002+00:00
3   1970-01-01 00:00:00.000000003+00:00
4   1970-01-01 00:00:00.000000004+00:00
dtype: datetime64[ns, UTC]
```

@jreback From my point of view, since Series accept `dtype='datetime64[ns, UTC]'`, why can't DataFrame accecpt it as well?

``` Python
array_d1 = np.arange(5)
array_d2 = np.arange(10).reshape((5, 2))
series = pd.Series(array_d1, dtype='datetime64[ns, UTC]')  # this works
df = pd.DataFrame(array_d2, dtype='datetime64[ns, UTC]') # doesn't work
```

that last is a bug, can you make a separate issue.

for now, let's exclude this from this PR, just need to finish this up.

@jreback Please review.

@jreback ping

pls rebase on master, make fixes asap.

@jreback rebased. please cancel the previous build because it will fail for bad code style..

looks good. ping when green.

@jreback ping

thanks @BranYang 

ok I added https://github.com/jreback/pandas/commit/b4cd033d0a2c39d98712ca22593f06a129a28e8a

basically the test weren't covering datetimelikes correctly as count was implemented (but not tested), while the other functions actually (sometimes) worked, incorrectly. This generates a complete set of tests (very fast). xref to #12535 which need to update the tests to account for nan values (they work) in count (so can test for datetimelikes with nans).

merging shortly.

thanks @BranYang