BUG: Add type promotion support for eval() expressions with many properties
State: closed

This commit modifies the call to numpy.result_type to get around the
NPY_MAXARGS limit, which at the moment is 32.  Instead of passing a
generator of all types involved in an expression, the type promotion
is done on a pair-wise basis with a call to reduce.

This fixes bugs for code such as the following:

```
from numpy.random import randn
from pandas import DataFrame

d = DataFrame(randn(10, 2), columns=list('ab'))

# Evaluates fine
print(d.eval('*'.join(['a'] * 32)))

# Fails to evaluate due to NumPy argument limits
print(d.eval('*'.join(['a'] * 33)))
```


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
If someone could chime in to just double check my understanding of the behavior of numpy.result_type, that would be nice.

I think this is an important fix, otherwise it is essentially impossible to evaluate expressions whose tree representation have more than 32 leaves.  Given that this limit seems imposed by NumPy, I assume it was not a conscious decision in the design of Pandas.

@havoc-io Gimme some time to review

I wasn't aware of this limit. Thanks for catching it.

No rush or anything on the review.  I want to make sure that my understanding of the result_type function (particularly in the context of the eval() infrastructure) is correct so this doesn't start returning some weird types from eval.

I've gone ahead and added a reduce import from pandas.compat as well and squashed it into this commit.

@havoc-io pls add the above test (which should fail without this change, and pass with it)

Importing from compat is fine

Added a test which fails without the change and succeeds with it.  The test will raise an exception under the old code.  Do you prefer to catch and then manually trigger failure - or just let the test framework catch the exceptions and mark them as E?

I've updated the commit.  The issue was marginally more extensive that I originally anticipated, so rather than scatter calls to `reduce` everywhere, I've added a `pandas.computation.common._result_type_many` utility function as a wrapper for calling `numpy.result_type` when passing an indefinite and potentially large number of arguments.  It also slightly changes the behavior of my previous code to ensure that the call correctly succeeds for argument lists of length 1.

I've left in-place calls to `numpy.result_type` which have a small, definite number of arguments, since the wrapper function is not necessary.

In addition, I've added the test recommended by @cpcloud (thanks!).

Another thing to check is the associativity and commutativity of result_type. If it's neither then using reduce isn't going to give the same result as result_type. Intuitively I would expect it to be both, akin to a sort of max operation on dtypes, but maybe there's some edge case that I'm not thinking of. 

That was my intuition as well, and does appear to be the case.  The NumPy result_type call stack (since NumPy 1.6) effectively looks like:
1. return_type (an alias for the C function [array_result_type](https://github.com/numpy/numpy/blob/b7850701a31127cad8c7399cea6be9cd5f71bec5/numpy/core/src/multiarray/multiarraymodule.c#L2988) which effectively just repackages the input array to be C-friendly), which calls
2. [PyArray_ResultType](https://github.com/numpy/numpy/blob/8358f04ce9e6651e79f83a5960f3e3fcd383538f/numpy/core/src/multiarray/convert_datatype.c#L1498) which calls
3. [PyArray_PromoteTypes](https://github.com/numpy/numpy/blob/8358f04ce9e6651e79f83a5960f3e3fcd383538f/numpy/core/src/multiarray/convert_datatype.c#L914)

The C code is a bit difficult to parse from a conceptional standpoint, but the [documentation](http://docs.scipy.org/doc/numpy/reference/c-api.array.html#PyArray_PromoteTypes) for the latter two functions gives a more concise explanation - which does indeed portray the functions as an effective "max" operation on dtypes.  `PyArray_PromoteTypes` even explicitly states that it is symmetric and associative.

I don't think it would be possible to write an exhaustive test to confirm this, but I think the algorithm described in the documentation is quite clearly symmetric and associative.  Aside from pinging the numpy developers, I'm not really sure how else to verify.

Okay, done, have a peek at the revised version.

looks ok 2 me ... i 'll merge when it passes

@havoc-io pls add a release notes (you will have to rebase as I just added the 0.14.0 section); add in bug fixes pls (you can ref this PR)

I've added an entry to the 0.14.0 bug fix log.

Looks like CI build passed - good to merge?

pls rebase on master; the release notes are in conflict

as soon as you do I can merge

Should be sorted now.  Thanks.

perfect thank you!

@havoc-io FYI, I had to move this to a class (rather than the global checker), as it didn't properly disable numexpr when necessary (e.g. when it was not available

https://github.com/pydata/pandas/commit/b0ba12ab5860900a2e3791355b159b19f9be4193

it works, but pls confirm that it is still correctly checking

Looks good to me.

gr8! this didn't show up till the windows builds started failing...they don't have numexpr installed at all