Patch for pandas.core.computation.eval with str/byte ndarrays
State: closed

Hello,
I am interested in using Pandas as a numexpr replacement, since its eval/query syntax is much cleaner. However, it doesn't quite work for me, since I rely on byte ndarrays, and I want to avoid DataFrames. Here is an example script that shows the problems I had, and how they are solved in a patched version. I am not very used to open source projects, I hope my patch will be acceptable.

cheers

Sjoerd

### Example script

```python
import numpy as np
import numexpr as ne
from pandas import DataFrame
from pandas.core.computation.eval import eval as pd_eval

# Sample data
s = np.zeros(6, 'S5')
v = np.array([5,8,3,6,7,2])
s[:4] = "Test", "Test2", "Test", "Test"
s_u = np.array([ss.decode() for ss in s])  # s as unicode
dic = {"s": s, "v": v}
dic_u = {"s": s_u, "v": v}


print("Golden standard (numexpr with ugly syntax)\t", ne.evaluate("(s == 'Test') & (v > 3)", dic))

df = DataFrame(dic)
print("df with bytes (wrong)\t\t\t\t", df.eval("s == 'Test' and v > 3").values)
try:  # ugly syntax AND does not work
    print(df.eval("s == b'Test' and v > 3").values)
except AttributeError:
    print("*" * 50)
    import traceback; traceback.print_exc(0)
    print("*" * 50)

df = DataFrame(dic_u)
print("df with unicode (works)\t\t\t\t", df.eval("s == 'Test' and v > 3").values)


from pandas.core.computation.eval import eval as pd_eval
print("pd_eval with bytes (wrong)\t\t\t", pd_eval("s == 'Test' and v > 3", global_dict=dic))
print("pd_eval with unicode (wrong)\t\t\t", pd_eval("s == 'Test' and v > 3", global_dict=dic_u))

print("patched pd_eval with bytes (works)\t\t", pd_eval("s == 'Test' and v > 3", global_dict=dic, align_result=False, str_as_bytes=True))
print("patched pd_eval with unicode (works)\t\t", pd_eval("s == 'Test' and v > 3", global_dict=dic_u, align_result=False, str_as_bytes=True))
```

### Output
```
Golden standard (numexpr with ugly syntax)       [ True False False  True False False]
df with bytes (wrong)                            [False False False False False False]
**************************************************
Traceback (most recent call last):
AttributeError: 'PandasExprVisitor' object has no attribute 'visit_Bytes'
**************************************************
df with unicode (works)                          [ True False False  True False False]
pd_eval with bytes (wrong)                       [False False False False False False]
pd_eval with unicode (wrong)                     [False False False False False False]
patched pd_eval with bytes (works)               [ True False False  True False False]
patched pd_eval with unicode (works)             [ True False False  True False False]
```

Labels:

Comments:
Hello @sjdv1982! Thanks for updating the PR.

 - There are no PEP8 issues in the file [`pandas/core/computation/engines.py`](https://github.com/sjdv1982/pandas/blob/45c7ac80e7af9cd8718a35a3479a4f46b03e6431/pandas/core/computation/engines.py) !

 - There are no PEP8 issues in the file [`pandas/core/computation/eval.py`](https://github.com/sjdv1982/pandas/blob/45c7ac80e7af9cd8718a35a3479a4f46b03e6431/pandas/core/computation/eval.py) !

 - There are no PEP8 issues in the file [`pandas/core/computation/expr.py`](https://github.com/sjdv1982/pandas/blob/45c7ac80e7af9cd8718a35a3479a4f46b03e6431/pandas/core/computation/expr.py) !



##### Comment last updated on December 11, 2018 at 16:27 Hours UTC
w/o tests this would not be acceptable. However, we do not deal with raw ndarray's at all as input here to ``.eval()``; so this is creaping out of scope. We have removed similar handling for all top-level pandas routines, so -1 on adding this back.
So If I understand correctly: for the Pandas project, ```core.computation.eval``` is meant for only DataFrames and Series in its global/local dict? If I want to use it for something else, I should spin it off?
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24228?src=pr&el=h1) Report
> Merging [#24228](https://codecov.io/gh/pandas-dev/pandas/pull/24228?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/c994e98c0e216622ad025c0c9e28421a4a3ee6af?src=pr&el=desc) will **decrease** coverage by `49.2%`.
> The diff coverage is `53.33%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24228/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24228?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #24228       +/-   ##
===========================================
- Coverage   92.21%      43%   -49.21%     
===========================================
  Files         162      162               
  Lines       51763    51769        +6     
===========================================
- Hits        47733    22264    -25469     
- Misses       4030    29505    +25475
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `43% <53.33%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24228?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/computation/eval.py](https://codecov.io/gh/pandas-dev/pandas/pull/24228/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vZXZhbC5weQ==) | `14.42% <0%> (-82.7%)` | :arrow_down: |
| [pandas/core/computation/engines.py](https://codecov.io/gh/pandas-dev/pandas/pull/24228/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vZW5naW5lcy5weQ==) | `44.61% <40%> (-44.28%)` | :arrow_down: |
| [pandas/core/computation/expr.py](https://codecov.io/gh/pandas-dev/pandas/pull/24228/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vZXhwci5weQ==) | `62.72% <66.66%> (-25.85%)` | :arrow_down: |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/24228/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24228/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/24228/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/24228/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/24228/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/24228/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-98.62%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24228/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-95.46%)` | :arrow_down: |
| ... and [122 more](https://codecov.io/gh/pandas-dev/pandas/pull/24228/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24228?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24228?src=pr&el=footer). Last update [c994e98...45c7ac8](https://codecov.io/gh/pandas-dev/pandas/pull/24228?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24228?src=pr&el=h1) Report
> Merging [#24228](https://codecov.io/gh/pandas-dev/pandas/pull/24228?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/c994e98c0e216622ad025c0c9e28421a4a3ee6af?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `86.66%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24228/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24228?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24228      +/-   ##
==========================================
- Coverage   92.21%   92.21%   -0.01%     
==========================================
  Files         162      162              
  Lines       51763    51769       +6     
==========================================
+ Hits        47733    47737       +4     
- Misses       4030     4032       +2
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.61% <86.66%> (-0.01%)` | :arrow_down: |
| #single | `43% <53.33%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24228?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/computation/eval.py](https://codecov.io/gh/pandas-dev/pandas/pull/24228/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vZXZhbC5weQ==) | `97.11% <100%> (ø)` | :arrow_up: |
| [pandas/core/computation/engines.py](https://codecov.io/gh/pandas-dev/pandas/pull/24228/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vZW5naW5lcy5weQ==) | `87.69% <80%> (-1.2%)` | :arrow_down: |
| [pandas/core/computation/expr.py](https://codecov.io/gh/pandas-dev/pandas/pull/24228/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vZXhwci5weQ==) | `88.43% <88.88%> (-0.14%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24228?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24228?src=pr&el=footer). Last update [c994e98...45c7ac8](https://codecov.io/gh/pandas-dev/pandas/pull/24228?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

> So If I understand correctly: for the Pandas project, core.computation.eval is meant for only DataFrames and Series in its global/local dict? If I want to use it for something else, I should spin it off?

not really sure what you are attempting to do in the first place.
> > So If I understand correctly: for the Pandas project, core.computation.eval is meant for only DataFrames and Series in its global/local dict? If I want to use it for something else, I should spin it off?
> 
> not really sure what you are attempting to do in the first place.

Instead of this (ugly):

```numexpr.evaluate("(s == 'Test') & (v > 3)", dic))```

I want this (clean):

```pd_eval("s == 'Test' and v > 3", dic)```


@sjdv1982 again I am not sure what you are actually trying to do. pls show a complete example.
> @sjdv1982 again I am not sure what you are actually trying to do. pls show a complete example.

[This](https://github.com/sjdv1982/struclib/blob/master/tests/viewer.ipynb). 

I am building a molecular viewer, and I want users to be able to use [clean selection syntax](https://pymolwiki.org/index.php/Selection_Algebra), no bitmasks.
closing. this is out of scope for pandas. all operations, by definition align. this is moving away from the core data structures.