spacetelescope/asv — pull request #99 by astrofrog (2014-09-13 22:24:33+00:00)
Title: Compare command

This is just a very experimental idea for a 'compare' command that can be used to compare two specific revisions, e.g:

```
$ asv compare 7810d6d7 19aa5743
· Fetching recent changes.

All benchmarks:

before after ratio
98.30ms 68.19ms 0.69 benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_ann_center
139.58ms 108.97ms 0.78 benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_ann_exact
98.94ms 67.96ms 0.69 benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_ann_subpixel_01
105.35ms 73.63ms 0.70 benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_ann_subpixel_05
116.30ms 86.23ms 0.74 benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_ann_subpixel_10
48.15ms 51.00ms 1.06 benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_center
68.18ms 69.50ms 1.02 benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_exact
48.59ms 51.24ms 1.05 benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_subpixel_01
51.13ms 53.71ms 1.05 benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_subpixel_05
56.26ms 58.65ms 1.04 benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_subpixel_10
4.71s 231.49ms 0.05 benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_elli_ann_center
1.70s 1.67s 0.98 benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_elli_ann_exact
4.72s 242.42ms 0.05 benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_elli_ann_subpixel_01
failed 454.41ms n/a benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_elli_ann_subpixel_05
failed 1.13s n/a benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_elli_ann_subpixel_10
...
```

Benchmarks are color-coded as green if they are faster by a given threshold, and red if they are slower. The threshold can be adjusted with the ``--threshold`` option.

The output can be split between benchmarks that got better, didn't change, or got worse, with the ``--split`` option:

```
$ asv compare 7810d6d7 19aa5743 --split -t 1.5
· Fetching recent changes..

Benchmarks that have improved:

before after ratio
4.71s 231.49ms 0.05 benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_elli_ann_center
4.72s 242.42ms 0.05 benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_elli_ann_subpixel_01
failed 454.41ms n/a
[snip]
failed 159.65ms n/a benchmarks.AperturePhotometry.time_small_data_multiple_small_apertures_elli_subpixel_05
failed 215.41ms n/a benchmarks.AperturePhotometry.time_small_data_multiple_small_apertures_elli_subpixel_10

Benchmarks that have stayed the same:

before after ratio
98.30ms 68.19ms 0.69 benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_ann_center
139.58ms 108.97ms 0.78 benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_ann_exact
98.94ms 67.96ms 0.69
[snip]
1.55ms 1.52ms 0.98 benchmarks.AperturePhotometry.time_small_data_single_small_aperture_elli_subpixel_05
1.72ms 1.58ms 0.92 benchmarks.AperturePhotometry.time_small_data_single_small_aperture_elli_subpixel_10

Benchmarks that have got worse:

before after ratio
2.30ms 3.71ms 1.61 benchmarks.AperturePhotometry.time_big_data_single_big_aperture_elli_center
2.31ms 3.71ms 1.61 benchmarks.AperturePhotometry.time_big_data_single_big_aperture_elli_subpixel_01
```

I implemented this because I needed it for a specific set of benchmarks where I'm trying to compare the impact of different optimization strategies, but of course you can reject this or make suggestions for significant changes, as needed.