ENH: Use numexpr for all arithmetic operators + refactor arithmetic operations
State: closed

Resolves #3765.
#### This is going to be split into two parts and then rebased on top of #3482.

Tasks:
- [x] Move arithmetic operations to separate file (e.g., `ops.py`).
- [x] Explore combining series/panel/frame operation methods.
- [x] Add rand, rxor, ror, etc...

This adds:
1. numexpr acceleration to `Series`, `Panel`, `Panel4D`, and `DataFrame` (`SparsePanel` has to opt-out b/c doesn't respond to shape)
2. Adds a test mechanism to numexpr to check that it actually is accelerated.
3. Moves all arithmetic operators to core/generic and binds them at import to reduce code duplication.
4. Makes all of `Series`, `Panel`, `Panel4D`, `DataFrame`, `SparsePanel` support the entire range of arithmetic operators [previously some were missing].
5. Cleans up core/expressions slightly
6. Test cases for all operations.
7. Improves a few test utilities to be better (error message for isnull and better error for isinstance checks in util/testing)

@cpcloud - I think this would be very easy to add into your current changes with eval. (mostly just changes `if hasattr` --> `getattr` to be cleaner).


Labels:

Comments:
I can combine a few of the commits if necessary ... just wanted to make the progression clearer.

This is great. Numexpr ftw. 

yay ... another network error :-/

fixes r on the way

moving at the speed of travis :|

well, I realized I missed testing the entire set of arithmetic methods in the individual cases (outside of test_expressions), so I guess it's okay.  Btw - is it worth it to mark the panel tests as slow in expressions if they add 1.5s to the total suite?

(1.5s _total_)

Is it a single test? If so yes else no 

no, it's not.

@jtratner this real nice..

not that I think ops actually make much sense on a `Panel4D`, but maybe throw a couple of tests, e.g. maybe adding them or something...?

@jreback okay, will do.

@jtratner 

@cpcloud and I created a new master branch to handle the new eval stuff (which is actually ready to merge, just after 0.13), branch is eval-3393 (and links to the eval PR); feel free to add any of this type of stuff to itt

oh awesome. I'll definitely do that.

On Fri, Jul 19, 2013 at 9:08 PM, jreback notifications@github.com wrote:

> @jtratner https://github.com/jtratner
> 
> @cpcloud https://github.com/cpcloud and I created a new master branch
> to handle the new eval stuff (which is actually ready to merge, just after
> 0.13), branch is eval-3393 (and links to the eval PR); feel free to add any
> of this type of stuff to itt
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/4051#issuecomment-21285701
> .

you can push, just DONT rebase
and make sure you pull before you push, if you can't push let me know

@jreback good to know.

@jtratner after I update #3482 / #4324 

I think it would be appropriate to use part of this PR, namely which moved the arithmetic/comp ops to generic.py to cover all of NDFrame (as Series will then inherit too), e.g. your `_add_flex....` type methods...(which I guess will live in `ops.py`?
(this is essentially the 3rd todo in #4324)

the numexpr stuff can be done separately (after)

Yeah, I agree with you on that. Adding  numexpr speedup is trivial compared
to normalizing ops between all the parts.

To be clear: do you want me to make a PR and/or push some of these changes
to eval-3393 branch that you have already or do you want me to wait until
after its merged?
On Jul 23, 2013 6:47 PM, "jreback" notifications@github.com wrote:

> @jtratner https://github.com/jtratner after I update #3482https://github.com/pydata/pandas/issues/3482/
> #4324 https://github.com/pydata/pandas/issues/4324
> 
> I think it would be appropriate to use part of this PR, namely which moved
> the arithmetic/comp ops to generic.py to cover all of NDFrame (as Series
> will then inherit too), e.g. your _add_flex.... type methods...(which I
> guess will live in ops.py?
> (this is essentially the 3rd todo in #4324https://github.com/pydata/pandas/issues/4324
> )
> 
> the numexpr stuff can be done separately (after)
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/4051#issuecomment-21451945
> .

I think that eval-3393 is pretty independent of the series refactor. what you can do if you want is make a new PR on top of series #3482 (or do after I have merged). The numexpr stuff let's do after both of these PR's are in in your artihmetic/comparison changes.

Makes sense

@jreback should probably merge series first then eval...i'll rebase after that merge and then test again to make sure things are working correctly....but shouldn't really matter that much

Now that I think about it, separating out the numexpr part will make this _way_ easier to apply on top of that PR.

Closing as per @jreback 's suggestion and will split into two separate PRs.