Repository: pandas-dev/pandas

Issue:
eval - 'Call' nodes are not implemented
df = P.DataFrame({'a':[1,1,2,2,3,3]})
df.eval("a.mean()")

raise:
NotImplementedError: 'Call' nodes are not implemented

i hope this will be possible
thank's 
Gla



Discussion:
What you've got there doesn't make a whole lot of sense (maybe you're doing something that does and this is just an example):

``` python
df = DataFrame({'a': [1, 1, 2, 2, 3, 3]})
m = df.a.mean()
df.eval('m') == m
pd.eval('m') == m
```

so just evaluate it in Python.

No new features will be added to `eval`, `query` for the next release (0.13) and possibly even 0.14

Slightly OT: @glaucouri Just in case that error message is a bit cryptic: A "Call" node is a node in the _abstract syntax tree_ created by parsing Python source. [Here](http://docs.python.org/2/reference/expressions.html#calls) is the language reference on callable objects.

What i'm trying to explain (with my poor english) is that in an eval environment if [a] is a P.Series i expect that i can use a this syntax : ( this is my working program, that users can use for do personalized equations)

safe_env = dict( [(k,df[k]) for k in df.keys() ])
safe_env # some other injections  from numpy
val = eval(equation,safe_env)
#now val is the result of the equation, so with a little of broadcasting i can insert it in the df

Actually, with this solution users can do filtering, grouping and equation in a DataFrame environment.

Thank's
Gla

@glaucouri What you've just described is essentially what happens under the hood in `DataFrame.query()`/`DataFrame.eval()`. Currently, you cannot call functions with either of those methods. Does that make sense?

Closing as stale ... not really a demand for this right now 

Out of curiosity, would a (well-written) pull request adding support for the numexpr intrinsic functions be accepted?  I've noticed that there is a variable defined in the expression parser called `_numexpr_supported_calls`, which seems to enumerate these functions, but this variable doesn't seem to be used anywhere.  Stuff like `abs` or some of the transcendental functions might be quite useful.

we always accept a well-written PR (even some not so well-written!) :)

@havoc-io Yes, a PR would be great! To get started (if you haven't already) take a look in `pandas/computation/expr.py`. This is where the parsing logic is implemented. You'd need to allow `Call` nodes by removing them from the set exclusion happening in the decorator.

Here's an example of what a `Call` node looks like for the code `f(a, b, c=1)`:

``` python
import ast
from ast import *  # forgive the star import here
node = ast.parse('f(a, b, c=1)', mode='eval').body
node.func == Name(id='f', ctx=Load())
node.args == [Name(id='a', ctx=Load()), Name(id='b', ctx=Load())]
node.keywords == [keyword(arg='c', value=Num(n=1))]
```

You could probably skip implementing keyword arguments for now and just implement unary or binary functions to get a feel for how difficult (or not) it would be to do this.

Eish, this may be a bit complicated actually, given the conditional handling of `Call` nodes (not to mention their subnodes).  It might bloat the parsing code quite a bit, maybe more than it's worth.  Perhaps simply transforming `Series` objects before `eval`ing is going to be easier for now for my use case.

It may also not be as well-defined as I initially imagined.  E.g., would you want the behavior for only the 'numexpr' engine or the 'python' engine as well (with appropriate replacement functions from math or NumPy)?  I'm also not sure what the behavior should be if `local_dict` overrides one of the intrinsics - I'd have to see what numexpr does in this case.

Maybe best to leave this issue dead for now :(.

@havoc-io There are quite a few details to be worked out here, which is part of the reason I didn't implement these right away.

It _would_ be nice to have the `python` engine for testing and speed comparisons, but you could also just test using regular `numpy` based, toplevel python code. Maybe see how hard it would be to do the simplest thing.
- only `numexpr` for now
- add the funcs to the global namespace (just for validation) ... this may require some kind of custom "delayed" call node that doesn't do anything because you don't want to actually evaluate it

I believe numexpr overrides intrinsics so a variable called sin wouldn't work. 


Resolving patch:
diff --git a/pandas/computation/expr.py b/pandas/computation/expr.py
--- a/pandas/computation/expr.py
+++ b/pandas/computation/expr.py
@@ -20,7 +20,7 @@
                                     _arith_ops_syms, _unary_ops_syms, is_term)
 from pandas.computation.ops import _reductions, _mathops, _LOCAL_TAG
 from pandas.computation.ops import Op, BinOp, UnaryOp, Term, Constant, Div
-from pandas.computation.ops import UndefinedVariableError
+from pandas.computation.ops import UndefinedVariableError, FuncNode
 from pandas.computation.scope import Scope, _ensure_scope
 
 
@@ -524,27 +524,48 @@ def visit_Call(self, node, side=None, **kwargs):
         elif not isinstance(node.func, ast.Name):
             raise TypeError("Only named functions are supported")
         else:
-            res = self.visit(node.func)
+            try:
+                res = self.visit(node.func)
+            except UndefinedVariableError:
+                # Check if this is a supported function name
+                try:
+                    res = FuncNode(node.func.id)
+                except ValueError:
+                    # Raise original error
+                    raise
 
         if res is None:
             raise ValueError("Invalid function call {0}".format(node.func.id))
         if hasattr(res, 'value'):
             res = res.value
 
-        args = [self.visit(targ).value for targ in node.args]
-        if node.starargs is not None:
-            args += self.visit(node.starargs).value
+        if isinstance(res, FuncNode):
+            args = [self.visit(targ) for targ in node.args]
+
+            if node.starargs is not None:
+                args += self.visit(node.starargs)
+
+            if node.keywords or node.kwargs:
+                raise TypeError("Function \"{0}\" does not support keyword "
+                                "arguments".format(res.name))
+
+            return res(*args, **kwargs)
+
+        else:
+            args = [self.visit(targ).value for targ in node.args]
+            if node.starargs is not None:
+                args += self.visit(node.starargs).value
 
-        keywords = {}
-        for key in node.keywords:
-            if not isinstance(key, ast.keyword):
-                raise ValueError("keyword error in function call "
-                                 "'{0}'".format(node.func.id))
-            keywords[key.arg] = self.visit(key.value).value
-        if node.kwargs is not None:
-            keywords.update(self.visit(node.kwargs).value)
+            keywords = {}
+            for key in node.keywords:
+                if not isinstance(key, ast.keyword):
+                    raise ValueError("keyword error in function call "
+                                     "'{0}'".format(node.func.id))
+                keywords[key.arg] = self.visit(key.value).value
+            if node.kwargs is not None:
+                keywords.update(self.visit(node.kwargs).value)
 
-        return self.const_type(res(*args, **keywords), self.env)
+            return self.const_type(res(*args, **keywords), self.env)
 
     def translate_In(self, op):
         return op
@@ -587,7 +608,7 @@ def visitor(x, y):
         return reduce(visitor, operands)
 
 
-_python_not_supported = frozenset(['Dict', 'Call', 'BoolOp', 'In', 'NotIn'])
+_python_not_supported = frozenset(['Dict', 'BoolOp', 'In', 'NotIn'])
 _numexpr_supported_calls = frozenset(_reductions + _mathops)
 
 
diff --git a/pandas/computation/ops.py b/pandas/computation/ops.py
--- a/pandas/computation/ops.py
+++ b/pandas/computation/ops.py
@@ -16,9 +16,12 @@
 
 
 _reductions = 'sum', 'prod'
-_mathops = ('sin', 'cos', 'exp', 'log', 'expm1', 'log1p', 'pow', 'div', 'sqrt',
-            'inv', 'sinh', 'cosh', 'tanh', 'arcsin', 'arccos', 'arctan',
-            'arccosh', 'arcsinh', 'arctanh', 'arctan2', 'abs')
+
+_unary_math_ops = ('sin', 'cos', 'exp', 'log', 'expm1', 'log1p',
+                   'sqrt', 'sinh', 'cosh', 'tanh', 'arcsin', 'arccos',
+                   'arctan', 'arccosh', 'arcsinh', 'arctanh', 'abs')
+_binary_math_ops = ('arctan2',)
+_mathops = _unary_math_ops + _binary_math_ops
 
 
 _LOCAL_TAG = '__pd_eval_local_'
@@ -498,3 +501,28 @@ def return_type(self):
             (operand.op in _cmp_ops_dict or operand.op in _bool_ops_dict)):
             return np.dtype('bool')
         return np.dtype('int')
+
+
+class MathCall(Op):
+    def __init__(self, func, args):
+        super(MathCall, self).__init__(func.name, args)
+        self.func = func
+
+    def __call__(self, env):
+        operands = [op(env) for op in self.operands]
+        return self.func.func(*operands)
+
+    def __unicode__(self):
+        operands = map(str, self.operands)
+        return com.pprint_thing('{0}({1})'.format(self.op, ','.join(operands)))
+
+
+class FuncNode(object):
+    def __init__(self, name):
+        if name not in _mathops:
+            raise ValueError("\"{0}\" is not a supported function".format(name))
+        self.name = name
+        self.func = getattr(np, name)
+
+    def __call__(self, *args):
+        return MathCall(self, args)


Test patch:
diff --git a/pandas/computation/tests/test_eval.py b/pandas/computation/tests/test_eval.py
--- a/pandas/computation/tests/test_eval.py
+++ b/pandas/computation/tests/test_eval.py
@@ -23,7 +23,8 @@
 from pandas.computation.expr import PythonExprVisitor, PandasExprVisitor
 from pandas.computation.ops import (_binary_ops_dict,
                                     _special_case_arith_ops_syms,
-                                    _arith_ops_syms, _bool_ops_syms)
+                                    _arith_ops_syms, _bool_ops_syms,
+                                    _unary_math_ops, _binary_math_ops)
 
 import pandas.computation.expr as expr
 import pandas.util.testing as tm
@@ -1439,6 +1440,129 @@ def setUpClass(cls):
         cls.arith_ops = expr._arith_ops_syms + expr._cmp_ops_syms
 
 
+class TestMathPythonPython(tm.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        super(TestMathPythonPython, cls).setUpClass()
+        tm.skip_if_no_ne()
+        cls.engine = 'python'
+        cls.parser = 'pandas'
+        cls.unary_fns = _unary_math_ops
+        cls.binary_fns = _binary_math_ops
+
+    @classmethod
+    def tearDownClass(cls):
+        del cls.engine, cls.parser
+
+    def eval(self, *args, **kwargs):
+        kwargs['engine'] = self.engine
+        kwargs['parser'] = self.parser
+        kwargs['level'] = kwargs.pop('level', 0) + 1
+        return pd.eval(*args, **kwargs)
+
+    def test_unary_functions(self):
+        df = DataFrame({'a': np.random.randn(10)})
+        a = df.a
+        for fn in self.unary_fns:
+            expr = "{0}(a)".format(fn)
+            got = self.eval(expr)
+            expect = getattr(np, fn)(a)
+            pd.util.testing.assert_almost_equal(got, expect)
+
+    def test_binary_functions(self):
+        df = DataFrame({'a': np.random.randn(10),
+                        'b': np.random.randn(10)})
+        a = df.a
+        b = df.b
+        for fn in self.binary_fns:
+            expr = "{0}(a, b)".format(fn)
+            got = self.eval(expr)
+            expect = getattr(np, fn)(a, b)
+            np.testing.assert_allclose(got, expect)
+
+    def test_df_use_case(self):
+        df = DataFrame({'a': np.random.randn(10),
+                        'b': np.random.randn(10)})
+        df.eval("e = arctan2(sin(a), b)",
+                engine=self.engine,
+                parser=self.parser)
+        got = df.e
+        expect = np.arctan2(np.sin(df.a), df.b)
+        pd.util.testing.assert_almost_equal(got, expect)
+
+    def test_df_arithmetic_subexpression(self):
+        df = DataFrame({'a': np.random.randn(10),
+                        'b': np.random.randn(10)})
+        df.eval("e = sin(a + b)",
+                engine=self.engine,
+                parser=self.parser)
+        got = df.e
+        expect = np.sin(df.a + df.b)
+        pd.util.testing.assert_almost_equal(got, expect)
+
+    def check_result_type(self, dtype, expect_dtype):
+        df = DataFrame({'a': np.random.randn(10).astype(dtype)})
+        self.assertEqual(df.a.dtype, dtype)
+        df.eval("b = sin(a)",
+                engine=self.engine,
+                parser=self.parser)
+        got = df.b
+        expect = np.sin(df.a)
+        self.assertEqual(expect.dtype, got.dtype)
+        self.assertEqual(expect_dtype, got.dtype)
+        pd.util.testing.assert_almost_equal(got, expect)
+
+    def test_result_types(self):
+        self.check_result_type(np.int32, np.float64)
+        self.check_result_type(np.int64, np.float64)
+        self.check_result_type(np.float32, np.float32)
+        self.check_result_type(np.float64, np.float64)
+        # Did not test complex64 because DataFrame is converting it to
+        # complex128. Due to https://github.com/pydata/pandas/issues/10952
+        self.check_result_type(np.complex128, np.complex128)
+
+    def test_undefined_func(self):
+        df = DataFrame({'a': np.random.randn(10)})
+        with tm.assertRaisesRegexp(ValueError,
+                                   "\"mysin\" is not a supported function"):
+            df.eval("mysin(a)",
+                    engine=self.engine,
+                    parser=self.parser)
+
+    def test_keyword_arg(self):
+        df = DataFrame({'a': np.random.randn(10)})
+        with tm.assertRaisesRegexp(TypeError,
+                                   "Function \"sin\" does not support "
+                                   "keyword arguments"):
+            df.eval("sin(x=a)",
+                    engine=self.engine,
+                    parser=self.parser)
+
+
+class TestMathPythonPandas(TestMathPythonPython):
+    @classmethod
+    def setUpClass(cls):
+        super(TestMathPythonPandas, cls).setUpClass()
+        cls.engine = 'python'
+        cls.parser = 'pandas'
+
+
+class TestMathNumExprPandas(TestMathPythonPython):
+    @classmethod
+    def setUpClass(cls):
+        super(TestMathNumExprPandas, cls).setUpClass()
+        cls.engine = 'numexpr'
+        cls.parser = 'pandas'
+
+
+class TestMathNumExprPython(TestMathPythonPython):
+    @classmethod
+    def setUpClass(cls):
+        super(TestMathNumExprPython, cls).setUpClass()
+        cls.engine = 'numexpr'
+        cls.parser = 'python'
+
+
 _var_s = randn(10)