ENH: fix eval scoping issues
State: closed

closes #5987
closes #5087

Relevant user-facing changes:

```
In [9]: df = DataFrame(randn(5, 2), columns=list('ab'))

In [10]: a, b = 1, 2

In [11]: # column names take precedence

In [12]: df.query('a < b')
Out[12]:
          a         b
0 -0.805549 -0.090572
1 -1.782325 -1.594079
2 -0.984364  0.934457
3 -1.963798  1.122112

[4 rows x 2 columns]

In [13]: # we must use @ whenever we want a local variable

In [14]: df.query('@a < b')
Out[14]:
          a         b
3 -1.963798  1.122112

[1 rows x 2 columns]

In [15]: # we cannot use @ in eval calls

In [16]: pd.eval('@a + b')
  File "<string>", line unknown
SyntaxError: The '@' prefix is not allowed in top-level eval calls, please refer to your variables by name without the '@' prefix

In [17]: pd.eval('@a + b', parser='python')
  File "<string>", line unknown
SyntaxError: The '@' prefix is only supported by the pandas parser
```
- [x] update query/eval docstrings/indexing/perfenhancing
- [x] make sure docs build
- [x] release notes
- [x] pytables
- [x] make the `repr` of `Scope` objects work or revert to previous version
- [x] more tests for new local variable scoping API
- [x] disallow (and provide a useful error message for) locals in expressions like `pd.eval('@a + b')`
- [x] Raise when your variables have the same name as the [builtin math functions that `numexpr` supports](https://code.google.com/p/numexpr/wiki/UsersGuide#Supported_functions), since you cannot override them in `numexpr.evaluate`, even when explicitly passing them. For example

``` python
import numexpr as ne
sin = randn(10)
d = {'sin': sin}
result = ne.evaluate('sin > 1', local_dict=d, global_dict=d)
result == array(True)
```

For reference, after this PR local variables are given lower precedence than column names. For example

``` python
a, b = 1, 2
df = DataFrame(randn(10, 2), columns=list('ab'))
res = df.query('a > b')
```

will no longer raise an exception about overlapping variable names. If you want the local `a` (as opposed to the column `a`) you must do

``` python
df.query('@a > b')
```


Labels:
name: Bug
name: API Design
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
oh boy! looks like the right way to do this..... :+1: 

Last issue is a scoping issue with py3 ... somehow not capturing scope from `sys._getlevel`.

guess this was not a small change :1234: 

Nope. But scope is less of a hack and I think overall implementation is easier to understand. Actually got up early to try and fix this but all the back and forth of comparing py2 and 3 is huge time sink

turns out that list comprehensions work differently re the stack in python 3....each iteration generates a new stack frame. std for loops don't work that way so that should do it

Ok this is passing locally on py27 and py33

@jreback Whenever you get a chance .... this is passing.

won't pretend I understand! hahah

can you build the docs and just check that everything looks ok their?

I find sometimes they are good tests in and of themselves (aside which they need slight changing anyhow, right?)

Oh right! Those have to be changed anyway becuase of the new API ... let me add some new check boxes..

Biggest changes are
- Use of `DeepChainMap` (a nice wrapper around dictionaries that lets you store/search/update multiple dictionaries in a particular order). This is the most major change and most important since it obviates a lot of my scope hacks that were there before.
- local API is slightly different more explicit as we discussed
- stronger syntax checks for locals and a helpful err msg

maybe post on the top the relevant changes (from an ipython session).....

sure

@y-p @jreback @jorisvandenbossche Would be very grateful for any comments y'all might have.

your exceptions should be multi line (looking at the syntax error one for using a @ in eval)

u should put what u put in release notes in v0.14 API change section

done

adding a test for #5087. this closes that too!

u create temporaries for the locals to avoid name collisions? 

looks fine to me
go ahead and squash

Temporaries are created for strings and lists

For locals I just rename them when the terms are created during parsing. 

oh ok

looks fine 2 me otherwise

can merge when u r ready

I am not really familiar with this code, but just added some minor doc comments.

Thanks for the comments guys.

@jreback Should I squash some more? This is ready for merging if there are no more comments.

maybe a bit more?

Okay squashed and cleaned up the query docstring. Would like to keep _some_ history so that I remember why I made change X especially some of these python version differences 

cool....go for it

bombs away