PERF: refactor tokenizer to give compiler branching hints. (OSX testers wanted)
State: closed

Included vbench shows ~12% improvement. I've seen 15-20% on movielens, 8% on FEC.
I've encountered no regressions on any dataset I tried.

Tested with gcc 4.6,4.7,4.8 on linux, results from OSX+clang would be useful: 
`./test_perf.sh -H -r read_table_multiple_long` before and after.

Second VB commit is unrelated.

Will merge in a couple of days or so, if all is quiet.


Labels:

Comments:
@y-p On your branch and master (at c1ab38e), when I try this:

``` shell
(pandas-dev)Toms-MacBook-Pro:pandas tom$ ./test_perf.sh -H -r read_table_multiple_long
readlink: illegal option -- f
usage: readlink [-n] [file ...]
python: can't open file '/vb_suite/test_perf.py': [Errno 2] No such file or directory
(pandas-dev)Toms-MacBook-Pro:pandas tom$ 
```

It looks like it's trying to run from the root of my drive, not relative to the current directory.
Am I running it incorrectly?

Nothing wrong that I see. try launching `vb_suite/test_perf.py` directly?
test_perf.sh is just there in the root to make the script more visible.

It's probably a BSD vs linux thing, `readlink -f` is not portable I would guess.

Edit: Should be fixed now.

Ahh, I'm hitting an error with vbench / my pytz build. From the traceback.

```
    TIME_ZONE = pytz.timezone('US/Eastern')
AttributeError: 'module' object has no attribute 'timezone'
```

I'll try to debug this, and then I'll report back with the tests.

edit: The readlink fix worked btw.

Could I be hitting an import from within a package issue here?  From my site-packages folder, `import pytz` works fine.  But from `pandas/vb_suite` I get the when I run `import pandas` or `import pytz` I get the traceback.

```
(pandas-dev)Toms-MacBook-Pro:vb_suite tom$ python test_perf.py -h
Traceback (most recent call last):
  File "test_perf.py", line 55, in <module>
    import pandas as pd
  File "/Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas/pandas/__init__.py", line 25, in <module>
    import pandas.core.config_init
  File "/Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas/pandas/core/config_init.py", line 4, in <module>
    from pandas.core.format import detect_console_encoding
  File "/Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas/pandas/core/format.py", line 7, in <module>
    from pandas.core.index import Index, MultiIndex, _ensure_index
  File "/Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas/pandas/core/index.py", line 10, in <module>
    import pandas.index as _index
  File "index.pyx", line 34, in init pandas.index (pandas/index.c:14959)
  File "/Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pytz/__init__.py", line 35, in <module>
    from pkg_resources import resource_stream
  File "/Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pkg_resources.py", line 74, in <module>
    import parser
  File "/Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas/vb_suite/parser.py", line 1, in <module>
    from vbench.api import Benchmark
  File "/Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/vbench/__init__.py", line 2, in <module>
    import vbench.log
  File "/Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/vbench/vbench/__init__.py", line 2, in <module>
    import vbench.log
  File "/Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/vbench/log.py", line 34, in <module>
    from vbench.config import is_interactive
  File "/Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/vbench/vbench/config.py", line 3, in <module>
    TIME_ZONE = pytz.timezone('US/Eastern')
AttributeError: 'module' object has no attribute 'timezone'
```

So on lines 498 and 499 you have the comment:     "# move away from the pandas root dit, to avoid possible import surprises"

But the import error (if that's actually the problem) happens before that.  I just hardcoded that to my home directory and got the same error.

Does python's import mechanism remember where a program/the interpreter is launched from?

Starting from somewhere outside the pandas repo I can import pandas just fine.  But starting from `pandas/vb_suite` and running

``` python
import os
os.chdir('/Users/tom')

import pandas as pd
```

raises the same error.

I'm not seeing any of these cases but you can always try using a virtualenv.
There's no need to spend too much time fighting your box for this test though, 
it's not that big of a deal.

Odd… I'm already running it inside a virtualenv.  I might have some time later to poke around with it.

Somewhat related note: are there plans to push something like `tets_perf` up to the vbench repo? I thought I read that in an issue but I'm not finding it.  I ask because I'm setting up a vbench suite for [statsmodels](https://github.com/statsmodels/statsmodels/pull/1070), and would be happy to help where I can.

Yeah, it's come up a few times. I don't plan to work on it soon myself, but for your benefit
(or anyone else interested) I put up an issue https://github.com/pydata/vbench/issues/34 with a summary of all the bits and pieces I'm aware of.
You're very welcome to take that on if it suits you, I have no problem ditching test_perf if 
upstream vbench gains that functionality.

If you want to discuss further, please do it there.

On it's own, this perf boost isn't significant enough to justify the extra code contortions. 
Maybe recycle as part of a bigger PR in the future.