Easy vbench testing of HEAD against a known-good baseline
State: closed

I hope to avoid more performance regressions in the future, in my own PR's
if no one else's... 

Her'es some example output:

```
groupby_apply_dict_return                 33.20      33.26  1.00
frame_reindex_axis1                        2.32       2.32  1.00
frame_fillna_inplace                      18.34      18.31  1.00
frame_ctor_nested_dict_int64             106.89     106.40  1.00
index_int64_union                         94.21      93.68  1.01
write_csv_standard                       309.77     307.62  1.01
groupby_multi_different_functions         16.72      16.60  1.01
frame_ctor_nested_dict                    75.67      74.62  1.01
groupby_multi_python                      54.00      53.18  1.02
frame_to_csv                             356.20     347.55  1.02
groupby_multi_cython                      21.21      20.63  1.03
groupby_indices                            9.22       8.95  1.03
frame_ctor_list_of_dict                   95.60      91.88  1.04
series_ctor_from_dict                      2.84       2.65  1.07
sort_level_zero                            6.46       6.00  1.08
groupby_simple_compress_timing            70.36      62.56  1.12

Columns: test_name | head_time [ms] | baseline_time [ms] | ratio

- a Ratio of 1.30 means HEAD is 30% slower then the Baseline.

Head [951d13b] : ENH: Add test_perf.sh, make vbench dead easy.
Baseline [bdbca8e] : BUG: fix borked data-copying iteritems performance affecting DataFrame and SparseDataFrame. close #2273


*** Results were also written to the logfile at '/home/user1/src/pandas/vb_suite.log'
```

benchmarks for which HEAD takes less then 1ms are dropped.
## Remaining issues

1) vb_suite/join_merge.py contains two tests named 'join_dataframe_index_single_key_bigger'
, easy enough to fix, but that might break existing dbs, i'm not sure, so I left it alone.

2) Runner seems to choke on refs which are not an ancestor of the current HEAD
this looks like a shortcoming in vbench.GitRepo.
example trace:

``` python
Traceback (most recent call last):
  File "test_perf.py", line 134, in <module>
    main()
  File "test_perf.py", line 94, in main
    runner._run_and_write_results(h_baseline)
  File "/usr/local/lib/python2.7/dist-packages/vbench/runner.py", line 87, in _run_and_write_results
    n_active_benchmarks, results = self._run_revision(rev)
  File "/usr/local/lib/python2.7/dist-packages/vbench/runner.py", line 119, in _run_revision
    need_to_run = self._get_benchmarks_for_rev(rev)
  File "/usr/local/lib/python2.7/dist-packages/vbench/runner.py", line 173, in _get_benchmarks_for_rev
    timestamp = self.repo.timestamps[rev]
  File "/home/user1/src/pandas/pandas/core/series.py", line 470, in __getitem__
    return self.index.get_value(self, key)
  File "/home/user1/src/pandas/pandas/core/index.py", line 692, in get_value
    raise e1
KeyError: 'bdbca8e'
```

3) redirecting stdout/stderr does no work, maybe because os.system is used
under the covers. It frustrated my efforts funnel the crud into a log file,
and keep the output readable.

4) it would be nice if vbench.db.BenchmarkDB.get_ref_result() returned a fully
 labeled DataFrame, rather then just a partial dict (test names are not included)

Something like:

``` python
    def get_rev_full_results(self, revision):
        tab = self._results
        bench = self._benchmarks

        stmt = sql.select([tab.c.timestamp, tab.c.revision, tab.c.ncalls,
                           tab.c.timing, tab.c.traceback, tab.c.description,
                           tab.c.checksum,bench.c.name],
                           sql.and_(tab.c.revision == revision,tab.c.checksum == bench.c.checksum))
        results = self.conn.execute(stmt)

        df = _sqa_to_frame(results).set_index('name')
        return df.sort_index()
```


Labels:

Comments:
This is awesome. I'll have to take a closer look but I'm kicking myself for not doing this myself by now-- good to have a quick sanity check before releasing to find any major perf regressions since the last release. 

changed to always redo the baseline test, so we're guranteed to pick up tests added since last run.

Cool. I will merge this in shortly. It would also be nice to be able to choose the benchmark hash or tag at the command line (e.g. compare v0.8.0 with master, etc.)

Note also that there are sub-1ms benchmarks in the suite with meaningful performance information. Though differences under 10% should be taken with a grain of salt

I'll add some argparse boilerplate soon.

I found the timings to vary quite a bit between runs, for quick measurements the random variation yields 
significant relative change and false positives which I found confusing. 
This was meant to simplify things for devs to get a quick red/green before opening a PR. 

Having said that,  I don't mind dropping that filter if you prefer.

I was thinking it should be possible to integrate something like this as a test for travis,
assuming their boxes have consistent performance between runs. Set a generous margin,
say 20%, and pass/fail based on the ratio.
We'll need to maintain a db with baseline results in tree, and hack something to retrieve the results of a 
baseline run on travis somehow to populate the db, but it looks doable.

FYI I just pushed the above commits to master

ping @wesm, af5e3d7 in my vbench branch adds commandline arguments.

**edit**: opened PR instead, with more fixes.

@y-p I identified a couple tests that were using up excessive amounts of memory, so you might be getting some variation because you're butting up against your swap space. Just a theory, anyway. I've been getting reasonably consistent results on my machine, thus far