API: raise TypeError on most datetime64 reduction ops
State: closed

closes #3726.


Labels:

Comments:
will squash when travis passes

@jreback i'm getting line ending problems kind of across the board. tox doesn't like when things end with dos line endings, but gh says to use `git config --global core.autocrlf input` which preserves the input, but if i run `dos2unix` on pandas then there 122 files that get converted which seems like it's asking for merge conflicts...i'm a little confused about what to do

I have crlf set to true which means ignore any endings
(I actually edit in windows) though run everything on Linux

ok it's no big deal i will just stop messing with tox*.ini

rebased

@jreback this is good 2 go if u want 2 merge

ok.....why don't you change it to 0.11.1....
also can you do perf check to make sure nothing changes...e.g.

test_perf -b <commit before this branch> -t <your last commit>

and see if anything >1.15 comes ups (that's related)

getting an error from that script...

i'll fix ti

there's a star import since the parser change that wesm made that overrides the dateutil parser

what's the error?

i fixed

that shouldn't matter, it pulls from git

[this](https://github.com/pydata/vbench/blob/master/vbench/git.py#L7) comes after [this](https://github.com/pydata/vbench/blob/master/vbench/git.py#L1) or just look at the first seven lines of https://github.com/pydata/vbench/blob/master/vbench/git.py

in `_parse_commit_log` there's a `from vbench.git import parser` which imports the pandas parser, i'll submit a pr to vbench since star imports generally lead to strange bugs. i've already tracked one of these down in some cython in pandas in the past. it was a hellish soup of `git bisect` and lots of coffee!

maybe i will also submit a pr to pandas that allows one to use the git syntax for commit hashes. i always have to git log at least twice to memorize the short hash number...

i see some that are > 1.15, but i see those in other vbench runs as well with the same test name. let me investigate...

post the bottom of the chart

```
frame_reindex_both_axes_ix                   |   0.3493 |   0.3234 |   1.0801 |
timeseries_add_irregular                     |  17.6566 |  16.2873 |   1.0841 |
frame_add_st                                 |   4.4290 |   4.0510 |   1.0933 |
series_constructor_ndarray                   |   0.0117 |   0.0106 |   1.0970 |
groupby_indices                              |   6.9420 |   6.0651 |   1.1446 |
reindex_multiindex                           |   1.2194 |   1.0020 |   1.2169 |
mask_floats                                  |   3.7626 |   2.9873 |   1.2595 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
```

that is my most recent commit (which I haven't pushed, just a doc update) vs. master

you have numexpr installed right? numpy 1.7.1?

```
INSTALLED VERSIONS
------------------
Python: 2.7.5.final.0
OS: Linux 3.9.4-1-ARCH #1 SMP PREEMPT Sat May 25 16:14:55 CEST 2013 x86_64
LC_ALL: None
LANG: en_US.UTF-8

Cython: 0.20dev
Numpy: 1.8.0.dev-e9e490a
Scipy: 0.13.0.dev-e2c502f
statsmodels: 0.5.0.dev-917da60
    patsy: 0.1.0+dev
scikits.timeseries: 0.91.3
dateutil: 2.1
pytz: 2013b
PyTables: 3.0.0
    numexpr: 2.1.1.dev
matplotlib: 1.4.x
openpyxl: 1.6.2
xlrd: 0.9.3dev
xlwt: 0.7.5
sqlalchemy: 0.8.1
lxml: 3.2.1
bs4: 4.2.1
html5lib: 0.95-dev
```

prob just random....try mask_floats in ipython a couple of times...

can you put up a %prun of doing it?

sure thing need a little time...

take your time

```
In [18]: paste
data = np.random.randn(10000, 500)
df = pd.DataFrame(data)
df = df.where(df > 0) # create nans
bools = df > 0
## -- End pasted text --

In [19]: mask = isnull(df)

In [20]: timeit bools.astype(float).mask(mask)
timeit 10 loops, best of 3: 40.7 ms per loop

In [21]: timeit bools.astype(float).mask(mask)
10 loops, best of 3: 40.6 ms per loop

In [22]: timeit bools.astype(float).mask(mask)
10 loops, best of 3: 41.2 ms per loop
```

the prun

```
         485 function calls in 0.047 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.024    0.024    0.024    0.024 necompiler.py:667(evaluate)
        2    0.009    0.004    0.009    0.004 {method 'astype' of 'numpy.ndarray' objects}
        1    0.004    0.004    0.004    0.004 {method 'ravel' of 'numpy.ndarray' objects}
        1    0.004    0.004    0.004    0.004 {operator.inv}
        3    0.003    0.001    0.003    0.001 {method 'copy' of 'numpy.ndarray' objects}
        1    0.001    0.001    0.039    0.039 frame.py:5581(mask)
        1    0.000    0.000    0.047    0.047 <string>:1(<module>)
        2    0.000    0.000    0.000    0.000 {method 'reduce' of 'numpy.ufunc' objects}
        4    0.000    0.000    0.040    0.010 internals.py:1243(apply)
        7    0.000    0.000    0.000    0.000 internals.py:928(make_block)
        7    0.000    0.000    0.000    0.000 internals.py:985(__init__)
        5    0.000    0.000    0.000    0.000 internals.py:1227(_verify_integrity)
        5    0.000    0.000    0.000    0.000 internals.py:1223(shape)
        2    0.000    0.000    0.009    0.004 common.py:1614(_astype_nansafe)
        7    0.000    0.000    0.000    0.000 frame.py:387(__init__)
        7    0.000    0.000    0.000    0.000 internals.py:1354(_consolidate_check)
        2    0.000    0.000    0.009    0.004 internals.py:250(astype)
        1    0.000    0.000    0.030    0.030 internals.py:498(where)
        1    0.000    0.000    0.029    0.029 internals.py:544(func)
        7    0.000    0.000    0.000    0.000 internals.py:39(__init__)
       68    0.000    0.000    0.000    0.000 {isinstance}
        1    0.000    0.000    0.035    0.035 frame.py:5523(where)
        1    0.000    0.000    0.004    0.004 frame.py:889(__invert__)
        7    0.000    0.000    0.000    0.000 generic.py:516(__init__)
        2    0.000    0.000    0.000    0.000 frame.py:1747(as_matrix)
        1    0.000    0.000    0.000    0.000 expressions.py:58(_can_use_numexpr)
        3    0.000    0.000    0.003    0.001 internals.py:134(copy)
        1    0.000    0.000    0.025    0.025 expressions.py:111(_where_numexpr)
        1    0.000    0.000    0.000    0.000 necompiler.py:462(getContext)
       65    0.000    0.000    0.000    0.000 {len}
        5    0.000    0.000    0.000    0.000 generic.py:668(_consolidate_inplace)
        2    0.000    0.000    0.009    0.004 generic.py:529(astype)
        1    0.000    0.000    0.000    0.000 fromnumeric.py:2098(prod)
       18    0.000    0.000    0.000    0.000 {issubclass}
        5    0.000    0.000    0.000    0.000 frame.py:2109(__setattr__)
       23    0.000    0.000    0.000    0.000 index.py:2715(_ensure_index)
        2    0.000    0.000    0.002    0.001 internals.py:1481(copy)
        4    0.000    0.000    0.000    0.000 {numpy.core.multiarray.array}
        4    0.000    0.000    0.000    0.000 numeric.py:273(asarray)
        1    0.000    0.000    0.000    0.000 internals.py:2199(create_block_manager_from_blocks)
       10    0.000    0.000    0.000    0.000 {hasattr}
       17    0.000    0.000    0.000    0.000 internals.py:1173(_get_items)
        1    0.000    0.000    0.001    0.001 frame.py:3392(fillna)
        2    0.000    0.000    0.000    0.000 common.py:1556(is_datetime64_dtype)
        1    0.000    0.000    0.000    0.000 frame.py:531(_init_ndarray)
        2    0.000    0.000    0.000    0.000 numerictypes.py:735(issubdtype)
        2    0.000    0.000    0.000    0.000 generic.py:37(_get_axis_number)
        7    0.000    0.000    0.000    0.000 internals.py:78(set_ref_locs)
        1    0.000    0.000    0.000    0.000 _methods.py:32(_all)
        5    0.000    0.000    0.000    0.000 generic.py:593(_clear_item_cache)
        1    0.000    0.000    0.000    0.000 frame.py:564(_wrap_array)
        1    0.000    0.000    0.000    0.000 {method 'all' of 'numpy.ndarray' objects}
        6    0.000    0.000    0.000    0.000 internals.py:1634(_consolidate_inplace)
        1    0.000    0.000    0.002    0.002 frame.py:2583(reindex)
        2    0.000    0.000    0.009    0.004 internals.py:1299(astype)
       10    0.000    0.000    0.000    0.000 internals.py:1229(<genexpr>)
        1    0.000    0.000    0.002    0.002 frame.py:2690(_reindex_multi)
       15    0.000    0.000    0.000    0.000 internals.py:1225(<genexpr>)
        1    0.000    0.000    0.001    0.001 frame.py:2717(_reindex_columns)
        5    0.000    0.000    0.000    0.000 {sum}
        6    0.000    0.000    0.000    0.000 frame.py:588(_constructor)
        1    0.000    0.000    0.000    0.000 frame.py:5736(_prep_ndarray)
        2    0.000    0.000    0.000    0.000 common.py:1566(is_timedelta64_dtype)
        1    0.000    0.000    0.001    0.001 frame.py:2724(_reindex_with_indexers)
        1    0.000    0.000    0.000    0.000 frame.py:584(axes)
        1    0.000    0.000    0.001    0.001 internals.py:1293(fillna)
        1    0.000    0.000    0.030    0.030 internals.py:1275(where)
        1    0.000    0.000    0.000    0.000 {sorted}
        7    0.000    0.000    0.000    0.000 {method 'append' of 'list' objects}
        2    0.000    0.000    0.000    0.000 common.py:1516(is_integer)
        2    0.000    0.000    0.000    0.000 internals.py:1499(as_matrix)
        1    0.000    0.000    0.000    0.000 frame.py:592(shape)
        5    0.000    0.000    0.000    0.000 {getattr}
        3    0.000    0.000    0.000    0.000 index.py:923(reindex)
        2    0.000    0.000    0.000    0.000 numerictypes.py:667(issubclass_)
        3    0.000    0.000    0.000    0.000 necompiler.py:616(getType)
        1    0.000    0.000    0.001    0.001 frame.py:2638(reindex_axis)
       11    0.000    0.000    0.000    0.000 internals.py:1346(is_consolidated)
        7    0.000    0.000    0.000    0.000 internals.py:130(dtype)
        1    0.000    0.000    0.000    0.000 _methods.py:24(_prod)
        2    0.000    0.000    0.000    0.000 {sys._getframe}
        5    0.000    0.000    0.000    0.000 internals.py:1620(consolidate)
        4    0.000    0.000    0.000    0.000 internals.py:122(shape)
        1    0.000    0.000    0.002    0.002 generic.py:858(copy)
        1    0.000    0.000    0.025    0.025 expressions.py:164(where)
        6    0.000    0.000    0.000    0.000 frame.py:467(_init_mgr)
        1    0.000    0.000    0.000    0.000 {zip}
        1    0.000    0.000    0.000    0.000 internals.py:1359(is_mixed_type)
        1    0.000    0.000    0.000    0.000 internals.py:295(_try_coerce_result)
        3    0.000    0.000    0.000    0.000 index.py:1412(equals)
        1    0.000    0.000    0.000    0.000 internals.py:291(_try_coerce_args)
        5    0.000    0.000    0.000    0.000 {method 'clear' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'items' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'copy' of 'dict' objects}
        6    0.000    0.000    0.000    0.000 {method 'pop' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 generic.py:695(_is_mixed_type)
        1    0.000    0.000    0.001    0.001 internals.py:212(fillna)
        7    0.000    0.000    0.000    0.000 {method 'get' of 'dict' objects}
        4    0.000    0.000    0.000    0.000 {callable}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
```

that's about right...so prob just random (this sometimes happens with ne btw.....you can get differening perf on a vbench depending if you are running something else on the machine or even randomness)

took less time than i thought :)

well chrome is a process/memory hog and i have 10 tabs open so maybe i should close them b4 running vbench otherwise just gvim and tmux running