Incorporate a vbench run into Travis-CI
State: closed

A seperate 2.7 env is spawned, which is allowed to fail without
failing the entire build. The results are posted to a public gist as 
a json file, and the gist url is printed in the log.

A helper function in vb_suite/perf_HEAD.py accepts a travis
build number, It then locates the VBENCH job within the build, fetches the log 
through the Travis API, extract the gist raw_url from the log, fetches the results
as json and converts it into  a pandas dataframe.

It remains to be seen whether the results are consistent
enough between builds over time to actually make this useful.

Here's how to gather the results from a few consecutive test builds I ran:

``` python
%cd vb_suite

import pandas as pd
from perf_HEAD import get_build_results
builds=[3393087,3393105,3393122,3393125,3393130]
dfs=[get_build_results(x) for x in builds]
dfs2=[x[['timing']] for x in dfs]
for df,b in zip(dfs2,builds):
    df.columns=[str(b)]
df = dfs2[0]
for other in dfs2[1:]:
    df=df.join(other,how='outer')
df[:20]

                                   3393087     3393105     3393122     3393125     3393130
name                                                                                      
append_frame_single_homogenous    0.641988    0.340924    0.524198    0.400884    0.373756
append_frame_single_mixed         2.159144    1.354964    1.639278    1.421034    1.301402
concat_series_axis1             131.367922   56.542015   92.097044   75.879002   71.159506
ctor_index_array_string           0.009129    0.004975    0.005761    0.006048    0.007652
dataframe_get_value               0.001885    0.001201    0.001403    0.001305    0.001331
dataframe_getitem_scalar          0.005543    0.003445    0.003394    0.003432    0.003307
dataframe_reindex_columns         0.494599    0.233001    0.370013    0.334528    0.316710
dataframe_reindex_daterange       0.683607    0.329319    0.552313    0.506124    0.439592
datamatrix_getitem_scalar         0.004519    0.003790    0.003355    0.003436    0.003317
datetimeindex_add_offset          0.439327    0.227953    0.327404    0.410341    0.312527
datetimeindex_normalize          73.160005   51.629400   60.980701   59.092712   58.182502
frame_boolean_row_select          0.769411    0.596314    0.723754    0.564437    0.653430
frame_constructor_ndarray         0.064187    0.028872    0.033200    0.033250    0.042343
frame_ctor_list_of_dict         140.382051   86.547899  114.851952  127.656937  114.904165
frame_ctor_nested_dict          124.281883   73.726416   97.840071  106.689930  111.264944
frame_ctor_nested_dict_int64    229.266882  130.098104  143.503904  183.116913  152.652025
frame_fancy_lookup                2.752171    2.173202    2.223320    1.820600    2.148161
frame_fancy_lookup_all          135.908127  113.823605  117.978096  108.165979  113.741159
frame_fillna_inplace             25.430799   16.801000   20.938993   24.916101   21.809912
frame_fillna_many_columns_pad    23.508501   16.926599   20.899606   19.706178   17.867088
```


Labels:

Comments:
There's also a script utilizing `memory_profiler` to measure the memory requirements of individual
vbenchmarks, But I found no interesting patterns so I'm not sure it's reliable.

This is really cool. I merged and pushed now. I'm going to see about fixing the current travis failures. Hard to say how reliable the perf benchmarks will be between runs since probably different EC2 machines will be running them