Repository: pandas-dev/pandas

Issue:
String comparison in query()
Hi, it seems that string comparisons aren't supported in query() yet, so maybe this isn't a bug yet. Anyway, hopefully this behavior will be fixed for future editions of pandas.

``` python
import pandas as pd
import numexpr as ne

a = list('abcdef')
b = range(6)
df = pd.DataFrame({'X':pd.Series(a),'Y': pd.Series(b)})

df_Y = df.query('Y < 3')             ## Works fine.
ne_works = ne.evaluate('"a" < "d"')  ## ne_works == np.array([True])
df_X = df.query('X < "d"')           ## RuntimeError: max recursion depth exceeded
```



Discussion:
cc @cpcloud 

IIRC this is not implemented ATM

Yep only `==` and `!=` are tested for strings.

Let me take a look at why this happens and I'll clean up the error message or I'll implement it.

Interesting ... it only fails on the `numexpr` engine ... yay for writing tests first!

Ok this was simply a case of making those evaluate in Python space when object dtypes are being compared....

I think maybe a doc note about the fact that expressions that are evaluated on a case by case basis in terms of the types of their result is needed ... I could've sworn I wrote something about this, but maybe just in our conversations ....

hahah...

@chuyelchulo Couple of details that might interest you:
- Expressions that would result in an object dtype (including simple variable evaluation) have to be evaluated in Python space. The main reason is for back compat with older versions of numpy that will truncate a string if you call `astype(str)` on them if they have elements with more than 60 chars. We can't pass `object` arrays to `numexpr` thus string comparisons are evaluated in Python space.
- The upshot is that that _only_ applies to strings. So, if you have an expression for example, that's a string comparison `and`-ed together with another boolean expression that's from a numeric comparison, the numeric comparison will be evaluated by `numpexpr`. In fact, in general, `query`/`eval` will "pick out" the subexpressions that are `eval`-able by `numexpr` and those that must be evaluated in Python space transparently to the user.

I'm adapting that for the docs, PR coming shortly


Resolving patch:
diff --git a/pandas/computation/expr.py b/pandas/computation/expr.py
--- a/pandas/computation/expr.py
+++ b/pandas/computation/expr.py
@@ -508,7 +508,8 @@ def _possibly_eval(self, binop, eval_in_python):
 
     def _possibly_evaluate_binop(self, op, op_class, lhs, rhs,
                                  eval_in_python=('in', 'not in'),
-                                 maybe_eval_in_python=('==', '!=')):
+                                 maybe_eval_in_python=('==', '!=', '<', '>',
+                                                       '<=', '>=')):
         res = op(lhs, rhs)
 
         if self.engine != 'pytables':


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -12841,6 +12841,25 @@ def test_query_with_nested_string(self):
         for parser, engine in product(PARSERS, ENGINES):
             yield self.check_query_with_nested_strings, parser, engine
 
+    def check_query_lex_compare_strings(self, parser, engine):
+        tm.skip_if_no_ne(engine=engine)
+        import operator as opr
+
+        a = Series(tm.choice(list('abcde'), 20))
+        b = Series(np.arange(a.size))
+        df = DataFrame({'X': a, 'Y': b})
+
+        ops = {'<': opr.lt, '>': opr.gt, '<=': opr.le, '>=': opr.ge}
+
+        for op, func in ops.items():
+            res = df.query('X %s "d"' % op, engine=engine, parser=parser)
+            expected = df[func(df.X, 'd')]
+            assert_frame_equal(res, expected)
+
+    def test_query_lex_compare_strings(self):
+        for parser, engine in product(PARSERS, ENGINES):
+            yield self.check_query_lex_compare_strings, parser, engine
+
 class TestDataFrameEvalNumExprPandas(tm.TestCase):
 
     @classmethod