CI: Check ASV for failed benchmarks
State: closed

closes #19518

xref #19104

Not sure if this should belong in `ci/lint.sh`, but testing the possibility of running the asv benchmarks on ci for failed benchmarks.

Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/19236?src=pr&el=h1) Report
> Merging [#19236](https://codecov.io/gh/pandas-dev/pandas/pull/19236?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/e8620abc12a4c468a75adb8607fd8e0eb1c472e7?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/19236/graphs/tree.svg?src=pr&token=eZ4WkYLtcO&width=650&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/19236?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #19236   +/-   ##
=======================================
  Coverage   91.62%   91.62%           
=======================================
  Files         150      150           
  Lines       48680    48680           
=======================================
  Hits        44603    44603           
  Misses       4077     4077
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.99% <ø> (ø)` | :arrow_up: |
| #single | `41.73% <ø> (ø)` | :arrow_up: |


------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/19236?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/19236?src=pr&el=footer). Last update [e8620ab...54727be](https://codecov.io/gh/pandas-dev/pandas/pull/19236?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Created `asv.sh`. I'm not too familiar where to enable the script in `travis.yml`. 
I pushed a commit to add this to travis. let's see what it does.
Hmm so the builds are returning `/home/travis/.travis/job_stages: line 57: ci/asv.sh: Permission denied`

Do I need to check in the file with execution permissions? https://stackoverflow.com/a/42187039
it should be chmod 755
Looks like travis is all green. 
I am not convinced this actually did anything. sure it installed but it doesn't say it even ran. 
```
23.33s$ ci/asv.sh
inside ci/asv.sh
Check for failed asv benchmarks
Collecting git+https://github.com/spacetelescope/asv
  Cloning https://github.com/spacetelescope/asv to /tmp/pip-Dzkkcb-build
Requirement already satisfied: six>=1.4 in /home/travis/miniconda3/envs/pandas/lib/python2.7/site-packages (from asv==0.3.dev1207+7edf1903)
Installing collected packages: asv
  Running setup.py install for asv ... done
Successfully installed asv-0.3.dev1207+7edf1903
real	0m0.410s
user	0m0.361s
sys	0m0.048s
Check for failed asv benchmarks DONE
```
here was a run on my local system
```
real    21m8.666s
user    18m20.882s
sys     2m47.438s
```

so for sure this has to be a separate build.
Added the else statement. I also _think_ this may not be working because asv wants to know the machine configurations upon install, so I added an `asv machine --yes` statement. 
Here's the latest output from Travis:

```
$ ci/asv.sh
inside ci/asv.sh
Check for failed asv benchmarks
Collecting git+https://github.com/spacetelescope/asv
  Cloning https://github.com/spacetelescope/asv to /tmp/pip-eJJWMU-build
Requirement already satisfied: six>=1.4 in /home/travis/miniconda3/envs/pandas/lib/python2.7/site-packages (from asv==0.3.dev1207+7edf1903)
Installing collected packages: asv
  Running setup.py install for asv ... done
Successfully installed asv-0.3.dev1207+7edf1903
· No information stored about machine 'travis-job-pandas-dev-pandas-329321549.travisci.net'. I know about nothing.
            
I will now ask you some questions about this machine to identify it in the benchmarks.
1. machine:  A unique name to identify this machine in the results.
   May be anything, as long as it is unique across all the machines
   used to benchmark this project.  NOTE: If changed from the default,
   it will no longer match the hostname of this machine, and you may
   need to explicitly use the --machine argument to asv.
machine [travis-job-pandas-dev-pandas-329321549.travisci.net]: 2. os:  The OS type and version of this machine.  For example,
   'Macintosh OS-X 10.8'.
os [Linux 4.14.12-041412-generic]: 3. arch:  The generic CPU architecture of this machine.  For example,
   'i386' or 'x86_64'.
arch [x86_64]: 4. cpu:  A specific description of the CPU of this machine, including
   its speed and class.  For example, 'Intel(R) Core(TM) i5-2520M CPU
   @ 2.50GHz (4 cores)'.
cpu [Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz]: 5. ram:  The amount of physical RAM on this machine.  For example,
   '4GB'.
ram [61815180]: 
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
The build has been terminated
```

So it appears travis is running the build, but it thinks that there's a build issue because the ASV benchmarks take a while to run. 
ok I pushed to a separate build. if this works, then we would simply capture the output (as well as output to the screen). then grep it. you have to keep showing output otherwise things will timeout.
Thanks @jreback 

It looks like the alternative is to prefix the asv script execution with `- travis_wait 30 ci/asv.sh` to tell travis to wait for 30 minutes, or more if needed. 

https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
Darn, looks like the build is taking to long. It ran for 49 min 57 sec and output:

`The job exceeded the maximum time limit for jobs, and has been terminated.`

Looks like the max time for travis-ci.org jobs is 50 minutes. 

https://docs.travis-ci.com/user/customizing-the-build#Build-Timeouts
i restarted that job again, i think this should complete
Looks like the job hit the time limit again.

It looks like `ci/install_travis.sh` itself took ~11 minutes. I'm not sure if there anything in this script that can be skipped to allow more time for the asv script
I would not grep for the output yet, this obscure the progress; let's see how far it gets.
@jreback Looks like the build ran the asv within the time limit:

```
real	36m57.758s
user	32m53.399s
sys	3m42.420s
Check for failed asv benchmarks DONE
```
I will move forward with greping the output if all looks good. 
what changed to make this work?
I think the change from `asv run` to `asv dev` made this work. 

My next commit will be to grep the output and see if it still runs within the time limit of the build.
After adding a grep, received

```
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
```

Will now be adding a travis wait
So I think it works now? I guess I could also intentionally make one of the benchmarks wrong to see if this "exited with 1". Would appreciate a second look @jreback 

From the build:

```
$ travis_wait 40 ci/asv.sh
Still running (33 of 40): ci/asv.sh
The command ci/asv.sh exited with 0.
Log:
inside ci/asv.sh
Check for failed asv benchmarks
· No information stored about machine 'travis-job-pandas-dev-pandas-333531431.travisci.net'. I know about nothing.
            
I will now ask you some questions about this machine to identify it in the benchmarks.
1. machine:  A unique name to identify this machine in the results.
   May be anything, as long as it is unique across all the machines
   used to benchmark this project.  NOTE: If changed from the default,
   it will no longer match the hostname of this machine, and you may
   need to explicitly use the --machine argument to asv.
machine [travis-job-pandas-dev-pandas-333531431.travisci.net]: 2. os:  The OS type and version of this machine.  For example,
   'Macintosh OS-X 10.8'.
os [Linux 4.14.12-041412-generic]: 3. arch:  The generic CPU architecture of this machine.  For example,
   'i386' or 'x86_64'.
arch [x86_64]: 4. cpu:  A specific description of the CPU of this machine, including
   its speed and class.  For example, 'Intel(R) Core(TM) i5-2520M CPU
   @ 2.50GHz (4 cores)'.
cpu [Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz]: 5. ram:  The amount of physical RAM on this machine.  For example,
   '4GB'.
ram [61815180]: 
real	32m40.209s
user	29m11.933s
sys	3m24.802s
Check for failed asv benchmarks DONE
/home/travis/.travis/job_stages: line 169:  4585 Terminated              travis_jigger $! $timeout $cmd
The command "travis_wait 40 ci/asv.sh" exited with 0.
```
@jreback Okay I think this works now. The build failure for #35124.4 is unrelated, and the asv build "failed" because some of the rolling benchmarks for `cov` and `corr` took more than 60 seconds. 
looking good. a couple of things.

1) would catch future warnings for most things (you can use ``catch_warnings`` so its generic) as these are distracting
2) would tee to a file of the failed to a tmp file, then cat it at the end (and put some echos around this to make it prominent). So that we can see what exactly failed.
Warnings have been caught and now checking failed benchmarks through a file, `failed_asv.txt`

```
...
[100.00%] ··· Running ...a.TimedeltaProperties.time_timedelta_seconds     4.20μs
real	33m10.909s
user	29m41.485s
sys	3m37.056s
The following asvs benchmarks (if any) failed.
DONE displaying failed asvs benchmarks.
Check for failed asv benchmarks DONE
```
@mroeschke If it is not too difficult, can you do the clean-up of the benchmarks for rolling in a separate PR?
@jorisvandenbossche Added back the rolling methods. Sorry I saw your other comment too late and squashed my commits where I made edits for the rolling methods. Is it still possible to cherry pick from a squashed commit?
Ping @jorisvandenbossche any additional feedback? 
thanks @mroeschke 

let's see how this works. its the longest build, but ok for now.