PERF: add initial asv config and vbench->asv conversion script
State: closed

@jorisvandenbossche Here's the initial pass I was referencing in our discussion in #9660; also relevant closes #8361. 

A few caveats:
- The `eval` suite will have to be converted by hand, as my `ast` transformer isn't parsing the string in `eval("df1 + df2")`
- Lots of test failures that I haven't even looked at
  - `vbench` tended to swallow failing tests, so maybe this is okay?
- Uses `conda` instead of `virtualenv` as I'm on OSX and `pip install` fails on several packages.
- Takes a while - haven't had a chance to compare against `vbench`. It's possible `asv` is doing a lot more iterations by default.
- The conversion depends on a non-stdlib package called `astor`, which does `ast -> source` conversion. Not aware of a way to do this with the stdlib.
-  The code size blows up as I'm not de-duplicating the setup in any way. Probably something that will just have to be fixed by hand after the switch is made.

Still a work-in-progress, but hopefully good enough that you (or someone else) can give this a shot.


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
Comment on (4): `vbench` seems to have default target time of 1 sec, whereas in `asv` the default is longer `goal_time = 2`. You can override this by having `goal_time = 1` attribute in the benchmark classes. The second difference is that `asv` runs every benchmark in a separate process --- however, importing the benchmark suite takes some time (here probably ~ 0.5 sec), so running each benchmark takes at least that.

@qwhelan I think this would be really nice. Any progress on this?

@jreback Sorry about that - haven't had time for a month or so. There's a fair number of failing tests that need digging into (eg, one was using `globals()` to store test data, which breaks with my transformation). I'll look into these a bit more this weekend.

Just a remark: I don't know if you should put to much time in perfecting the vbench to asv conversion. We maybe rather should do the conversion once, and then some of the converted asv benches can be fixed manually without having to make to conversion perfect?

@jorisvandenbossche Yeah, of course - just want to make sure there are no obvious bugs with the translation, as it won't be easy to re-run once people start touching formatting/etc of the translated results.

@jreback @jorisvandenbossche Finally got around to doing the long-promised cleanup to this code. Currently in three commits:
- The first is a file that appears to be missing from the package creation. I believe it was being noticed because of the from-scratch-venvs that `asv` uses. Not sure why this wouldn't generally manifest. Let me know if this should be a separate PR.
- Second commit is various changes to the existing `vbench` tests. Some are just syntactical sugar changes to help my translation script work better, but most are fixing typos or errors that meant these tests never ran.
- Third commit is everything dealing with `asv` and as generated by my translation script. The existing errors under asv are almost entirely namespace related and will be easily fixable, just should probably wait to see if anyone desires making changes to the translation. `asv` is verbose about the errors, so they shouldn't go unnoticed like they would under `vbench`.

I think this is ready to merge if some failing tests is okay.

can you do something like this (for initial validation).

Compare a specific commit in master vs. 0.16.2 for both vbench and asv. Then compare these ratios. That way can see if any tests are left out (of either vbench or asv) and if anything is crazily different. 

Can you show how to run this (e.g. mini-instructions)?

how long does the entire suite take to run? are there min times for certain tests? eg. can tests be under 1ms? or is there some overhead 

I tested it a little bit, and it seems it will even work on windows! (big plus over vbench)

@jreback for the question about comparing commit against master, see http://asv.readthedocs.org/en/latest/commands.html#asv-continuous. An example of as scipy PR using that: https://github.com/scipy/scipy/pull/4712

The scipy example also show it is possible to have parameterized tests, so this is also something that can be used to reduce the duplication of code.

You can set an attribute `goal_time` to indicate how long one test should take (how many iterations are done). See first bullet points under 'Attributes' http://asv.readthedocs.org/en/latest/writing_benchmarks.html#timing

@jreback The default approach is a `goal_time` of `2s`, which I've reduced to `0.2s` here. There is an option to explicitly set the number of iterations, but that's not (currently) settable from the command line (nor is `goal_time`).

I probably won't be able to get to a comparison analysis until this weekend, at best.

@jorisvandenbossche Ah, awesome - I had not tried windows. `asv` also supports Python 3, but the translated `vbench` suite (unsurprisingly) looks like it'll need some work to get that running.

Alright, finally found some time and managed to get my new environment setup.

Here's the basic analysis:
- 637 benchmarks in total
  - 7 succeeding under asv but failing under vbench
  - 56 succeeding under vbench but failing under asv (expected due to translation issues)
- r^2 in the 97-98% range (t-stat of ~200).
  - slope and intercept are a bit off, but probably due to low number of samples in the vbench case (slope of 1.18 and y-intercept of -0.02s)

Here's the log-log plot of asv vs vbench:

![log_log_pandas](https://cloud.githubusercontent.com/assets/440095/9147529/038d4a66-3d1d-11e5-90c2-d8f2e0d0d37b.png)

Weirdly enough, the most noise is in the middle of the data set. I'm digging into whether there's a common cause behind those points having large errors (eg, network or disk access).

most benchmarks should be somewhere in the middle or left hand side( and if not should prob be adjusted) as we don't want this suite taking too long

the rhs prob need some adjustment (which we can do later) and/or might be io tests

@jreback so ~10ms as an ideal upper bound?

Here's the `.describe()` on the vbench results:

```
count    6.120000e+02
mean     2.455878e-01
std      1.443327e+00
min      1.000000e-07
25%      2.173575e-03
50%      8.596750e-03
75%      5.306438e-02
max      2.008107e+01
Name: 1, dtype: float64
```

Total of ~150s for a single run of everything; here are the slowest 10:

```
packers_write_sql                      2.923420
left_outer_join_index                  3.507703
groupby_ngroups_10000_pct_change       5.102321
groupby_ngroups_10000_mad              5.595383
groupby_ngroups_10000_value_counts     6.164015
panel_pct_change_minor                 9.942969
panel_pct_change_major                 9.978835
panel_pct_change_items                11.820503
groupby_ngroups_10000_describe        20.004133
replace_large_dict                    20.081070
Name: 1, dtype: float64
```

At least the `replace_large_dict` and `groupby_ngroups_10000_*` ones seem to originate from a performance regression noted at large `n`, so a bit unclear how much those can be trimmed down.

Well, usually < 1 s as an upper bound (on the slow part of the test). But usually they are 100ms ish.

doing a test run myself for kicks....:)

can you add a note to the Contributing docs [here](http://pandas.pydata.org/pandas-docs/stable/contributing.html#running-the-performance-test-suite) as well as a note in the whatsnew (you can just put a line in the highlites and then reference the contributing docs).

but looks good!

@qwhelan ok seemed to run ok for the 2.7 (failed on 3.4), not really sure why.

does this create its own conda environments somewhere?

how do I get a 'report' of the results?

maybe add some examples of how to do things in the docs (doesn't have to be this PR, but would be useful) once this is merged.

I think some files not getting cleaned up.

```
-rw-rw-r--   1 jreback  staff  40487624 Aug  7 20:12 __test__.h5
-rw-rw-r--   1 jreback  staff    562183 Aug  7 20:13 __test__.csv
-rw-rw-r--   1 jreback  staff   6402106 Aug  7 20:14 __test__.msg
```

@jreback Yeah, haven't looked into the 3.4 issues - I imagine there's some syntax that needs cleaning up and was never addressed as vbench only supported Python 2.

The conda envs are created in `asv_bench/envs/`. There's a small change I should make to the config to have it auto-detect conda or virtualenv for general use.

To get a webserver-based report, run:

```
asv publish
asv preview
```

The files not getting cleaned up are an issue in the original vbench benchmarks, they were just hidden in `vb_suite/`.

ok, looks great. I guess if we keep running them, then will make nice pretty graphs.

is their a way to get a summary of all?

go ahead and make the doc changes and can merge this.

then can follow up with other PR's. lmk when you push.

@jreback 

As for summaries, `asv continuous` automatically filters down to only those benchmarks that changed by more than some `factor` (defaults to `2`, configurable via `-f`).

```
BENCHMARK                                    BEFORE      AFTER     FACTOR
...compare.time_timestamp_series_compare    18.97ms     6.02ms   0.31755118x
...compare.time_series_timestamp_compare    19.05ms     6.03ms   0.31646644x
...index_slicers.time_multiindex_slicers    28.31ms     8.47ms   0.29923046x
...time_timedelta_convert_string_seconds   146.11ms    32.87ms   0.22494551x
...ies_isin_int64.time_series_isin_int64    14.38ms     3.18ms   0.22104606x
..._string.time_timedelta_convert_string   191.91ms    37.01ms   0.19287192x
...ta64_1.time_dtype_infer_timedelta64_1   162.89ms    27.19ms   0.16690464x
...t_categorical.time_concat_categorical      1.34s   122.99ms   0.09183963x
...timeseries_to_datetime_iso8601_format   155.70ms     4.80ms   0.03083291x

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
```

For a full table, `asv compare e118206d 912854` produces the following (after I patched what looks like a bug, I'll submit a PR to asv):

```
All benchmarks:

    before     after       ratio
  [e118206d] [912854  ]
   310.58ns   311.98ns      1.00  attrs_caching.getattr_dataframe_index.time_getattr_dataframe_index
     3.66μs     3.75μs      1.03  attrs_caching.setattr_dataframe_index.time_setattr_dataframe_index
    36.21ms    28.82ms      0.80  binary_ops.frame_add.time_frame_add
    15.84ms    15.09ms      0.95  binary_ops.frame_add_no_ne.time_frame_add_no_ne
    17.69ms    15.75ms      0.89  binary_ops.frame_add_st.time_frame_add_st
    66.89ms    66.18ms      0.99  binary_ops.frame_float_div.time_frame_float_div
    13.05ms    12.67ms      0.97  binary_ops.frame_float_div_by_zero.time_frame_float_div_by_zero
    43.16ms    43.30ms      1.00  binary_ops.frame_float_floor_by_zero.time_frame_float_floor_by_zero
    18.49ms    18.43ms      1.00  binary_ops.frame_float_mod.time_frame_float_mod
    13.65ms    13.65ms      1.00  binary_ops.frame_int_div_by_zero.time_frame_int_div_by_zero
    27.06ms    25.96ms      0.96  binary_ops.frame_int_mod.time_frame_int_mod
    36.30ms    34.25ms      0.94  binary_ops.frame_mult.time_frame_mult
    15.35ms    15.15ms      0.99  binary_ops.frame_mult_no_ne.time_frame_mult_no_ne
    16.03ms    15.77ms      0.98  binary_ops.frame_mult_st.time_frame_mult_st
    71.42ms    72.88ms      1.02  binary_ops.frame_multi_and.time_frame_multi_and
    48.75ms    48.68ms      1.00  binary_ops.frame_multi_and_no_ne.time_frame_multi_and_no_ne
    58.35ms    58.55ms      1.00  binary_ops.frame_multi_and_st.time_frame_multi_and_st
+    6.03ms    19.05ms      3.16  binary_ops.series_timestamp_compare.time_series_timestamp_compare
    49.67ms    58.85ms      1.18  binary_ops.timestamp_ops_diff1.time_timestamp_ops_diff1
    66.87ms    62.38ms      0.93  binary_ops.timestamp_ops_diff2.time_timestamp_ops_diff2
+    6.02ms    18.97ms      3.15  binary_ops.timestamp_series_compare.time_timestamp_series_compare
+  122.99ms      1.34s     10.89  categoricals.concat_categorical.time_concat_categorical
    35.05μs    35.17μs      1.00  ctors.ctor_index_array_string.time_ctor_index_array_string
    17.70μs    16.99μs      0.96  ctors.dtindex_from_series_ctor.time_dtindex_from_series_ctor
   122.82μs   122.66μs      1.00  ctors.frame_constructor_ndarray.time_frame_constructor_ndarray
    30.73μs    30.78μs      1.00  ctors.index_from_series_ctor.time_index_from_series_ctor
    22.79μs    23.87μs      1.05  ctors.series_constructor_ndarray.time_series_constructor_ndarray
     failed     failed       n/a  eval.eval_frame_add_all_threads.time_eval_frame_add_all_threads
     failed     failed       n/a  eval.eval_frame_add_one_thread.time_eval_frame_add_one_thread
     failed     failed       n/a  eval.eval_frame_add_python.time_eval_frame_add_python
     failed     failed       n/a  eval.eval_frame_add_python_one_thread.time_eval_frame_add_python_one_thread
     failed     failed       n/a  eval.eval_frame_and_all_threads.time_eval_frame_and_all_threads
     failed     failed       n/a  eval.eval_frame_and_python.time_eval_frame_and_python
     failed     failed       n/a  eval.eval_frame_and_python_one_thread.time_eval_frame_and_python_one_thread
     failed     failed       n/a  eval.eval_frame_chained_cmp_all_threads.time_eval_frame_chained_cmp_all_threads
     failed     failed       n/a  eval.eval_frame_chained_cmp_python.time_eval_frame_chained_cmp_python
     failed     failed       n/a  eval.eval_frame_chained_cmp_python_one_thread.time_eval_frame_chained_cmp_python_one_thread
     failed     failed       n/a  eval.eval_frame_mult_all_threads.time_eval_frame_mult_all_threads
     failed     failed       n/a  eval.eval_frame_mult_one_thread.time_eval_frame_mult_one_thread
     failed     failed       n/a  eval.eval_frame_mult_python.time_eval_frame_mult_python
     failed     failed       n/a  eval.eval_frame_mult_python_one_thread.time_eval_frame_mult_python_one_thread
     failed     failed       n/a  eval.query_datetime_index.time_query_datetime_index
     failed     failed       n/a  eval.query_datetime_series.time_query_datetime_series
     failed     failed       n/a  eval.query_with_boolean_selection.time_query_with_boolean_selection
     2.14ms     2.15ms      1.01  frame_ctor.frame_ctor_dtindex_BDayx1.time_frame_ctor_dtindex_BDayx1
     2.22ms     2.21ms      1.00  frame_ctor.frame_ctor_dtindex_BDayx2.time_frame_ctor_dtindex_BDayx2
     2.49ms     2.48ms      0.99  frame_ctor.frame_ctor_dtindex_BMonthBeginx1.time_frame_ctor_dtindex_BMonthBeginx1
     2.48ms     2.48ms      1.00  frame_ctor.frame_ctor_dtindex_BMonthBeginx2.time_frame_ctor_dtindex_BMonthBeginx2
     2.29ms     2.30ms      1.01  frame_ctor.frame_ctor_dtindex_BMonthEndx1.time_frame_ctor_dtindex_BMonthEndx1
     2.26ms     2.31ms      1.02  frame_ctor.frame_ctor_dtindex_BMonthEndx2.time_frame_ctor_dtindex_BMonthEndx2
     2.50ms     2.52ms      1.01  frame_ctor.frame_ctor_dtindex_BQuarterBeginx1.time_frame_ctor_dtindex_BQuarterBeginx1
...
```

The lines with `+` are the ones identified by the `factor` (and highlighted in red in my console).

@qwhelan an you make those updates to the contributing docs. Then can merge this.

In the meantime I also think a/some new vbench had been added, so that one will have to be converted as well.

@jreback how do you see the transition? Because for the current and new PRs we have to decide how to add new tests. As asv tests once this is merged? Or still vbenches? I would say to just go with asv (if the most essential things work?)

I think we could add this in. Then @qwhelan can followup with vbench -> asv conversions every once in a while (and IIRC refactor some of the current ones). Then we can switch over at some point.

We can certainly encourage new benches to be on asv however. That's why I'd like to be doc-ed (at least somewhat now).

I'm currently out of town so it'll be a couple of days (at best) before I
can regenerate the converted benchmarks and update the documentation. The
former should be straightforward as no refactoring has been done.
On Aug 18, 2015 4:41 PM, "Jeff Reback" notifications@github.com wrote:

> I think we could add this in. Then @qwhelan https://github.com/qwhelan
> can followup with vbench -> asv conversions every once in a while (and IIRC
> refactor some of the current ones). Then we can switch over at some point.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/9715#issuecomment-132391676.

Since there also new vbenches in current PRs, we will have to do an update with conversion of recently added vbenches anyway later on, so maybe that is not essential for now to merge this

@qwhelan what you have now is good. I'll tell you what. Will merge now. but can you do a follow up to:
- add docs (in contributing.md)
- add/update vbenches to current
- whatsnew note (put in highlites that this is added)

Yup, setting some reminders to follow up later this week.
On Aug 18, 2015 5:32 PM, "Jeff Reback" notifications@github.com wrote:

> @qwhelan https://github.com/qwhelan what you have now is good. I'll
> tell you what. Will merge now. but can you do a follow up to:
> - add docs (in contributing.md)
> - add/update vbenches to current
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/9715#issuecomment-132402988.