Updates for asv suite
State: closed

@jorisvandenbossche As you requested in #10849, here's the current state of affairs. The changes are:
- First patch:
  - Fix the translation script so that functions defined in the vbench `setup` string get defined as classmethods and called appropriately. This fixes most failing tests.
  - Some backwards-compatibility fixes around imports. Raising `NotImplementedError` inside `setup()` is equivalent to `SkipTest`, so I've tried to do that in a couple necessary cases (such as things using `test_parallel`)
  - Speaking of `test_parallel`, this required a bit of hackery to support not-existing. I implemented a no-op decorator by that name so that the actual definition of test cases wouldn't fail. We simply raise `NotImplementedError` rather than test without it, however.
- Second patch:
  - Support Python 3, the changes are essentially what I mentioned elsewhere.
  - `from pandas_vb_common import *` -> `from .pandas_vb_common import *`
  - Replace `xrange` with `range` and add `from pandas.compat import range` in `pandas_vb_common`
  - Replace `string.uppercase` with `string.ascii_uppercase`
- Third patch
  - `asv` instructions in `contributing.rst`, largely lifted from the vbench description. The current examples are minimalist, but most comparable to what vbench outputs.
- Fourth patch:
  - Re-run the translation script. I think some other additions may be included in the first patch due to having run it a few times while making changes.

cc @jreback @TomAugspurger 


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
@jreback regarding your comments, I think that mainly are the things that will have to be handled manually (@qwhelan is that correct?) That is the clean-up mentioned in #10849

sure that is fine
yes iirc that is a further cleanup

ok so merge when ready then

@jorisvandenbossche Yes, that is correct. The only manual changes were in `vbench_to_asv.py` and inside `vb_suite` (which were propagated to `asv_bench` by rerunning the former). I also manually ensured some new `asv` tests added by others didn't get overwritten.

Just pushed some small changes (StringIO change and added a new line in contributing). I think it's good to merge.

@qwhelan Thanks a lot for this!

Of course.

I also have a bunch of asv run data I should throw up on a github.io page
so the EuroScipy folks can look into some of the regressions (won't get to
this for probably 6+ hours).
On Aug 29, 2015 3:42 PM, "Joris Van den Bossche" notifications@github.com
wrote:

> @qwhelan https://github.com/qwhelan Thanks a lot for this!
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/10928#issuecomment-136058593.

@jorisvandenbossche My run data should be accessible here: http://qwhelan.github.io/pandas_asv/ .

An explicit list of regressions can be seen here: http://qwhelan.github.io/pandas_asv/#/regressions

`from pandas_vb_common import *` -> `from .pandas_vb_common import *`

has broken `test_perf.sh`. running benchmarks fails, with `ImportError: No module named pandas_vb_common`. (on python 2)

Now we did the conversion (and it is the idea we shouldn't have to do this again, as we start changing the asv benchmarks direclty), we can change this (`from pandas_vb_common import *`) back to how it was in the vb_suite benchmarks.