WIP: add top-level eval function
State: closed

closes #3393.

This is a ways from being mergeable but I thought I would put up the PR for people to play with and to get some other eyes on it. Tire-kick at will. A consolidated list of things to be completed to come.
# `eval` Roadmap
## Documentation
- [ ] enhancing performance section
- [x] `eval` docstring
- [ ] Warn users that this is _not_ a magic bullet
  - E.g., `x = 3; y = 2; pd.eval('x // y', engine='python')` is 1000 times slower than the same operation in actual Python.
## Engines
- [x] python engine for testing (basically a restricted eval, should make sure nothing dangerous a la `eval('os.system("sudo rm -rf /")')`) is possible here
- [x] numexpr engine
- [x] pytables engine (#4155)

Should python be the default engine for small frames?
## Functions
- [x] toplevel `isexpr`
- [x] move `eval` to top-level pandas
## Error Handling
- [x] syntax error handling
- [x] name error handling
## Alignment
- Algorithm
  1. flatten the tree of terms
  2. resolve the variables, special casing only numpy arrays/scalars/single unary operations (probably room for more  abstraction here)
  3. align by joining on the indices using the current broadcasting behavior
  4. update the original namespace with the newly aligned version of the underlying numpy array
  5. reconstruct the final object with the aforementioned joined indices and type
- [ ] punt to `'python'` engine when a `Series` and `DataFrame` both have `DatetimeIndexes` (I would rather not do this and just enforce the to-be-deprecated index-aligning-if-both-indices-are-datetimeindex behavior)
- [ ] `PeriodIndex`es don't work (well, they won't pass my tests) because of a stack overflow bug when joining a `DatetimeIndex` and a `PeriodIndex` (that bug is slated for 0.13, but a design decision needs to be made)
- [ ] alignment for `NDFrame` (is this really necessary?)
- [x] unary alignment
- [x] binary alignment
## Scope
- [x] deal with global scoping issues (not sure if this is tested well enough...which means probably not)
- [x] allow `Expr` objects to be passed to `eval` (`Expr` will pull in the local and global variables from the calling stack frame)
## Miscellaneous
- [ ] replace current `evaluate` function
- [ ] `Expr` in `NDFrame.__getitem__` cc @jreback
- [x] reconstruct the final object to return the expected output type, e.g., `df + df2` should return a `DataFrame` if `df` and `df2` are both `DataFrame` objects.
- [x] change `Expression` to `Expr`
- [x] add `truediv` keyword to `Expr`, default to `True`
## Operators
- [x] `%`
- [x] `**`
- [x] unary ops don't play well when they are operands of binary ops
## Tests
- [ ] `DatetimeIndex` and `PeriodIndex` objects are untested because of a few join and alignment issues.
- [ ] rewrite a few tests in the python evaluator to make sure the `//` operator works as it should (it works, but there may be edge cases that I haven't though of yet)
- [ ] `//` `numexpr` only supports scalar evaluation here: variables are not allowed so this will always eval in python space (half-assed it a bit in the tests here need to fix that)
- [x] truediv keyword tests
- [x] add tests for nans
- [x] fix `**` associativity issue in testing
## Near Future
- allow `and`, `or` and `not` aliases for the operators `&`, `|` and `~`, respectively
- attribute accessing e.g., `df.A` syntax
- reductions (`sum` and `prod`)
- math ops (sin, cos, tan, sqrt, etc.)
## Far Future
- non-python indexing (or just punt to python)


Labels:

Comments:
quick pass I didn't see a doc-string on eval (its prob there...but I didn't delve too deep)

also a quick writeup for enhancedperf section?

just a couple of examples

@jreback 

question:
1. i think using `eval` under the hood for pytables evaluation should replace all of the various incarnations of expressions [here](http://pandas.pydata.org/pandas-docs/stable/io.html#querying-a-table)...is that ok?

result being that only strings are allowed (which will be converted to `Expr` and passed to an evaluating function).

going to take much longer to finish this up to support all of those expression forms, but if it needs to be done that's ok too

for a string expression, that is exactly right (its conceptually simpler, because only scalars can be on the rhs in any expression), however also need to allow something like this:

`E('index>',Timestamp('20130101'))`

where `E` is replace `Term` and is a top-level that you evaluate

unless you think that you can evaluate:

`index>Timestamp('20130101')` ?

I think that the current `Term` should be a sub-class of `Expr` that just spits out a `numexpr` evaluable string (and does not actually evaluate it)

ok i think i'm starting to see the bigger picture here. `convert_value` and `TermValue` will be superceeded by `eval`,

in this code in `Selection.select`

``` python
return self.table.table.readWhere(self.condition, start=self.start, stop=self.stop)
```

`self.condition` should be generated by `engine.convert()`

is that right?

and `queryables` is basically an environment internal to the table that has for example the `index` part of `index > 20130101`

that's about right.

in `Selection` I am just using a list of `Terms` that are anded together

`queryables` is used for validation on the `lhs` of the expressions (its analagous to you looking up a variable in the enclosing/global scope, e.g. `df>1` if `df` is not defined then you raise, same here

and no alignment necessary, just some conversions on `rhs` terms

and `self.condition` is generated by `engine.convert()` correct

`self.condition` is equiv to this:

`pd.eval('index>Timestamp("20130101")',queryables=queryables,engine='pytables')`

cool maybe this will be easier than i thought...to the pytables-mobile!

advertising!

http://stackoverflow.com/questions/17390886/how-to-speed-up-pandas-multilevel-dataframe-sum/17394064#17394064

might be a good benchmark.....and maybe add this 'example' to the `EnhancingPerformance` section?

nice! thanks. i gave u +1

just an update on the `and` `or` `not` deal

essentially:

what you want to write if u like python

``` python
pd.eval('not x and y and z or w')
```

if you want to be terse

``` python
pd.eval('~(x & y & z | w)')
```

what the engine actually sees (you could write this too, if you're practicing typing parens fast w/o looking!)

``` python
pd.eval('~((((x) & (y)) & (z)) | (w))')
```

and of course u can mix and match if you like partial clarity + terseness

what's nice is that the python precedence is preserved, e.g., `not x and y` rewrites to `~((x) & (y))` (`not` binds loosely) whereas `~x and y` rewrites to `'(~(x)) & (y)'` (`~` binds tightly)

for reference: [operator precedence](http://docs.python.org/2/reference/expressions.html#operator-precedence)

if u want i can give more details, but essentially the `repr` of each op is `'({self.lhs}) {self.op} ({self.rhs})'.format(self=self)` and this happens recursively.

the ast manip is to "rewrite" the `ast.BoolOp` nodes as their respective `Bit*` node (really i'm just saying, interpret this as a bitwise op), but the python `ast` takes care of precedence for u so the precedence semantics remain.

here was the trickiest part

``` python
def visit_BoolOp(self, node):
    op = self.visit(node.op)
    def visitor(x, y):
        try:
            lhs = self.visit(x)
        except TypeError:
            lhs = x

        try:
            rhs = self.visit(y)
        except TypeError:
            rhs = y

        return op(lhs, rhs)

    operands = node.values
    return reduce(visitor, operands)
```

key part is the last two lines which because the python parser treats boolean expressions similar to how lisp does you have to reconstruct the `(and x1 x2 ... xN)` style op into `(and (... (and (and x1 x2) ...) ...) xN)`

I would have an engine option, maybe parsing=strict as the default
if you want to support python booleans then could pass parsing=extended or something like that
I don't think we want to turn this on by default even though it looks pretty - 

ok. there's not really any automatic way to test that though since `and`, `or` and `not` are not part of the `operator` module, i suppose python's `eval` could work though (for testing only)

I would put it on back burner anyhow (i know u did it)
but too much of a chnge right now 

sounds good, not worth too much time.

the pytables changes are quite daunting. i know those are most important for this to be merged, still sifting through how it works...reading tests and trying to understand what the classes do...which ones are important etc...

plus the issues with dt and period indexes

i think after those two then this might warrant some serious tire kicking

I think i'm going to allow callable parsing in the `'pytables'` engine for now, as it will make the transition to using `eval` and friends under the hood much easier. I'll just subclass the current parser something like `TableExprVisitor` and overload the `visit_Call` method...

i think in 0.14 (not a typo) that `Term` should be removed...deprecate in 0.13 in favor of what I'm calling `TableExpr` which is a subclass of `Expr` designed to work with `HDFStore` et al.

@jreback should arbitrary expressions be allowed with `'pytables'`?

using `filter` will be easy if only binary ops are allowed, but will be more complicated if arbitrary exprs are allowed

nvm. should allow them.

whatever is parsable by ne should be allowed
in reality though only simple expressions are prob needed

@jreback let me see if i have the right idea about how selections are computed

assume that we:
- have an `AppendableFrameTable` instance
- want the whole table (i.e., not iterating over chunks)
1. `HDFStore.select` is called
2. `select` calls `AppendableFrameTable.read`
3. `read` calls `read_axes` which sets the `selection` attribute
4. `selection` is then used in `process_axes` to make the selection using `filter`s (if any)

i think my plan of attack will be:
1. make a subclass of my `Term` class, maybe `TableTerm` which has the `filter` attribute and possibly a few other unexposed helper methods if necessary
2. the `Term` class in `pytables.py` will become something like `TableExpr` (a subclass of `Expr`) which is a lightweight container for `TableTerm`s (just like `Expr`)

I think I might be able to do this without touching any of the `Storer` code.

too complicated....going to futz around with your branch....give me a few

ok. seems like there's a lot going on in `pytables`. it's not clear to me how to incorporate the filters into my current scheme since there's no state regarding the operator except its name, e.g., `self.op = '!='`. i suppose i could special case two classes for the filterable operators to hold a filter...

a filter is just an expression that we are going to evaluation after the results come back (e.g. its a dimension that pytables doesn't select on), e.g. when you store a frame, the rows are stored, but the columns are stored as a single block), so selecting on the columns is basically just a reindex

ok....if I have an expression like `columns=A`, how can I tell the node visitor (which is now a sub-class of your existing one), to really make this `columns==A` ? (e.g. it interprets it as an Assign, but I want a different op (aside from actually rewriting it, I basically want to map `=` and `==` to the same thing

i think this should work

``` python
def visit_Assign(self, node):
    cmpr = ast.Compare(ops=[ast.Eq()], left=self.visit(node.targets[0]),
                        comparators=self.visit(node.value))
    return self.visit(cmpr)
```

you also need to remove the raise i put in to disallow statements

or add `ast.Assign` to the instance check in `visit_Module`

ok that won't work exactly yet, give me a sec ill make it work an push and put a link

dont' worry....i think i got what i need for a min

actually....need to transform an Assign node to a Compare node...would be helpful..?