Repository: pandas-dev/pandas

Issue:
numexpr 2.3.1 error with pandas 0.13.1
I just installed numexpr 2.3.1 with pandas 0.13.1 and got the following error:

  File "C:\Python27\lib\site-packages\pandas\core\ops.py", line 496, in wrapper
    arr = na_op(lvalues, rvalues)
  File "C:\Python27\lib\site-packages\pandas\core\ops.py", line 443, in na_op
    raise_on_error=True, *_eval_kwargs)
  File "C:\Python27\lib\site-packages\pandas\computation\expressions.py", line 176, in evaluate
    *_eval_kwargs)
  File "C:\Python27\lib\site-packages\pandas\computation\expressions.py", line 104, in _evaluate_numexpr
    *_eval_kwargs)
  File "C:\Python27\lib\site-packages\numexpr\necompiler.py", line 738, in evaluate
    NumExpr(ex, signature, *_context)
  File "C:\Python27\lib\site-packages\numexpr\necompiler.py", line 554, in NumExpr
    precompile(ex, signature, context)
  File "C:\Python27\lib\site-packages\numexpr\necompiler.py", line 498, in precompile
    ast = typeCompileAst(ast)
  File "C:\Python27\lib\site-packages\numexpr\necompiler.py", line 163, in typeCompileAst
    % (ast.value + '_' + retsig+basesig))
NotImplementedError: couldn't find matching opcode for 'mul_bbb'



Discussion:
and what exactly are you doing when it gives that error?

Good point.

```
n_nonzero_returns_idc = np.sum(portfolio_data[field_return_daily_idc].notnull() *
                               (portfolio_data[field_return_daily_idc] != 0.0))
```

why not just do `&` instead?

can't use booleans in `numexpr` with multiply; need to `astype(int)` if you reall y want to do that

as @cpcloud says, why are you not using `&`?

still ... should this fall back onto python/bottleneck ops?

& worked - had to change it everywhere. cpcloud is right: & is better than *. I started using & instead of \* when I realized that. May be it's a good reason to correct those things. Still, has numexpr become more hard ass about things like that?

maybe not catching the `NotImplementedError`?

and not testing with 2.3.1...hmmm will fix that...

I take that back...we ARE testing on window 2.3.1.....guess don't have a test for that particular case.....

@cpcloud when you have a chance?

i don't think numexpr has ever supported arithmetic ops on bool dtype arrays. not sure why this is throwing from pandas tho ... @jreback sure

used to work with numexpr 2.0.1 and pandas 0.12

I agree....should just catch and revert to python

@bluefir what numpy are you using?

looks like 2.1 changed this (to a warning then later deprecated)....see here: https://github.com/pydata/numexpr/blob/master/RELEASE_NOTES.rst

oh didn't know they were on gh awesom

now we can add opcodes and datetime timedelta support hurrah!!!

numpy 1.8.1

Ok, just to intercept here. Are you guys saying there is a faster way of doing what I am trying to do?

looks like you are trying to index something else but keep shape?

`df.where(cond1 & cond2)` 

or just select

`df[cond1 & cond2]`

to just get the indexed data

could also do `np.sum(df[the_slice_or_index].fillna(0.0) != 0)`

I just want to calculate the number of values in a column that are not null and not equal to zero.

`s[s!=0].count()`

Ha! Indeed. Thanks!

@jreback should the arith bool ops return the same thing as combine? combine actually does each operation "manually" then puts the result in an array of the same type as the lhs and rhs ... which makes `truediv` different for numpy arrays vs. combine ... just not sure what direction the consistency should go np or pandas

dividing bools is weird and is an edge case so it's not THAT important i just need to pick one

I think you just raise on all bool dtypes that are not boolean types (e.g. `&,|,~|-`). no reason for numexpr to support `*` which is the only meaningful one (`+` and `/`) don't make sense at all and subtraction is just negation


Resolving patch:
diff --git a/pandas/computation/expressions.py b/pandas/computation/expressions.py
--- a/pandas/computation/expressions.py
+++ b/pandas/computation/expressions.py
@@ -154,6 +154,20 @@ def _where_numexpr(cond, a, b, raise_on_error=False):
 set_use_numexpr(True)
 
 
+def _has_bool_dtype(x):
+    try:
+        return x.dtype == bool
+    except AttributeError:
+        return 'bool' in x.blocks
+
+
+def _bool_arith_check(op_str, a, b, not_allowed=frozenset(('+', '*', '-', '/',
+                                                           '//', '**'))):
+    if op_str in not_allowed and _has_bool_dtype(a) and _has_bool_dtype(b):
+        raise NotImplementedError("operator %r not implemented for bool "
+                                  "dtypes" % op_str)
+
+
 def evaluate(op, op_str, a, b, raise_on_error=False, use_numexpr=True,
              **eval_kwargs):
     """ evaluate and return the expression of the op on a and b
@@ -170,7 +184,7 @@ def evaluate(op, op_str, a, b, raise_on_error=False, use_numexpr=True,
                          return the results
         use_numexpr : whether to try to use numexpr (default True)
         """
-
+    _bool_arith_check(op_str, a, b)
     if use_numexpr:
         return _evaluate(op, op_str, a, b, raise_on_error=raise_on_error,
                          **eval_kwargs)


Test patch:
diff --git a/pandas/tests/test_expressions.py b/pandas/tests/test_expressions.py
--- a/pandas/tests/test_expressions.py
+++ b/pandas/tests/test_expressions.py
@@ -2,6 +2,7 @@
 # pylint: disable-msg=W0612,E1101
 
 import nose
+import re
 
 from numpy.random import randn
 
@@ -339,6 +340,24 @@ def testit():
         expr.set_numexpr_threads()
         testit()
 
+    def test_bool_ops_raise_on_arithmetic(self):
+        df = DataFrame({'a': np.random.rand(10) > 0.5,
+                        'b': np.random.rand(10) > 0.5})
+        names = 'add', 'mul', 'sub', 'div', 'truediv', 'floordiv', 'pow'
+        ops = '+', '*', '-', '/', '/', '//', '**'
+        msg = 'operator %r not implemented for bool dtypes'
+        for op, name in zip(ops, names):
+            if not compat.PY3 or name != 'div':
+                f = getattr(operator, name)
+                err_msg = re.escape(msg % op)
+
+                with tm.assertRaisesRegexp(NotImplementedError, err_msg):
+                    f(df, df)
+
+                with tm.assertRaisesRegexp(NotImplementedError, err_msg):
+                    f(df.a, df.b)
+
+
 if __name__ == '__main__':
     import nose
     nose.runmodule(argv=[__file__, '-vvs', '-x', '--pdb', '--pdb-failure'],
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -4761,13 +4761,13 @@ def _check_unary_op(op):
         _check_unary_op(operator.neg)
 
     def test_logical_typeerror(self):
-        if compat.PY3:
-            pass
-        else:
+        if not compat.PY3:
             self.assertRaises(TypeError, self.frame.__eq__, 'foo')
             self.assertRaises(TypeError, self.frame.__lt__, 'foo')
             self.assertRaises(TypeError, self.frame.__gt__, 'foo')
             self.assertRaises(TypeError, self.frame.__ne__, 'foo')
+        else:
+            raise nose.SkipTest('test_logical_typeerror not tested on PY3')
 
     def test_constructor_lists_to_object_dtype(self):
         # from #1074