pydata/pandas — pull request #2279 by y-p (2012-11-18 00:52:58+00:00)
Title: Easy vbench testing of HEAD against a known-good baseline

I hope to avoid more performance regressions in the future, in my own PR's
if no one else's...

Her'es some example output:
```
stats_rank_average_int 22.47 22.78 1.01
timeseries_add_irregular 23.57 23.95 1.02
panel_from_dict_all_different_indexes 152.66 155.40 1.02
series_align_int64_index 47.94 48.90 1.02
indexing_panel_subset 2.85 2.91 1.02
join_dataframe_index_multi 26.10 26.68 1.02
read_csv_thou_vb 38.81 39.99 1.03
frame_to_csv 348.21 360.96 1.04
index_int64_intersection 41.62 43.26 1.04
sort_level_zero 5.93 6.19 1.04
reindex_frame_level_align 1.21 1.27 1.04
timeseries_infer_freq 12.64 13.24 1.05
frame_fancy_lookup 1.98 2.07 1.05
groupby_simple_compress_timing 50.43 53.68 1.06
stats_rank_average 27.95 29.88 1.07
append_frame_single_mixed 2.17 2.35 1.09
join_dataframe_index_single_key_small 7.46 8.33 1.12

Columns: test_name | head_time [ms] | baseline_time [ms] | ratio

- Ratio of 1.0 means parity.
- Ratio of 1.30 means HEAD is 30% slower then the Baseline for that benchmark.

Head [3253ac4] : ENH: Add test_perf.sh, make vbench dead easy.
Baseline [bdbca8e] : BUG: fix borked data-copying iteritems performance affecting DataFrame and SparseDataFrame. close #2273

*** Results were also written to the logfile at '/tmp/pandas/vb_suite.log'
```

Remaining issues
---
1) vb_suite/join_merge.py contains two tests named 'join_dataframe_index_single_key_bigger'
, easy enough to fix, but that might break existing dbs, i'm not sure, so I left it alone.

2) Runner seems to choke on refs which are not an ancestor of the current HEAD
this looks like a shortcoming in vbench.GitRepo.
example trace:

```python
Traceback (most recent call last):
File "test_perf.py", line 134, in
main()
File "test_perf.py", line 94, in main
runner._run_and_write_results(h_baseline)
File "/usr/local/lib/python2.7/dist-packages/vbench/runner.py", line 87, in _run_and_write_results
n_active_benchmarks, results = self._run_revision(rev)
File "/usr/local/lib/python2.7/dist-packages/vbench/runner.py", line 119, in _run_revision
need_to_run = self._get_benchmarks_for_rev(rev)
File "/usr/local/lib/python2.7/dist-packages/vbench/runner.py", line 173, in _get_benchmarks_for_rev
timestamp = self.repo.timestamps[rev]
File "/home/user1/src/pandas/pandas/core/series.py", line 470, in __getitem__
return self.index.get_value(self, key)
File "/home/user1/src/pandas/pandas/core/index.py", line 692, in get_value
raise e1
KeyError: 'bdbca8e'
```

3) redirecting stdout/stderr does no work, maybe because os.system is used
under the covers. It frustrated my efforts funnel the crud into a log file,
and keep the output readable.

4) it would be nice if vbench.db.BenchmarkDB.get_ref_result() returned a fully
labeled DataFrame, rather then just a partial dict (test names are not included)

Something like:
```python
def get_rev_full_results(self, revision):
tab = self._results
bench = self._benchmarks

stmt = sql.select([tab.c.timestamp, tab.c.revision, tab.c.ncalls,
tab.c.timing, tab.c.traceback, tab.c.description,
tab.c.checksum,bench.c.name],
sql.and_(tab.c.revision == revision,tab.c.checksum == bench.c.checksum))
results = self.conn.execute(stmt)

df = _sqa_to_frame(results).set_index('name')
return df.sort_index()
```