Cython cache diff compare
State: closed

As talked about in #13425 with @gfyoung and @jreback this PR does cython caching by comparing the pyx files and not relying on the git history. 


Labels:
name: Build
description: Library building on various platforms

Comments:
At the moment this code does the follow:
1. Caches all the pyx files at the end of a run under `$HOME/.cache/pyxfiles` keeping the directory structure incase two cython files have the same name but in different directories. 
2. Caches all the corresponding compiled files for those pyx files.

then at the start of a run:
1. Finds all the current pyx files in the checkout git repo on travis
2. For each file compares it to the file that is in the cache
3. If the file does not exist in cache a new cython file has been added => Do not use cython cache and rebuild
4. If the file is different from the cython file in cache => Do not use cython cache and rebuild
5. If the number of pyx files are different then a file has been deleted (a rename will be picked up by 3) => Do not use cython cache and rebuild
6. If all files are the same => Use cython cache, i.e. extract and touch the compiled pyx files.

~~The thing this does not do is the case where a file exists on the cache but has been deleted from the repo. Do you think this should be caught as well or should it not matter? I guess a compiled file would be copied over but then just not referenced or used in the build.~~

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13526?src=pr) is **84.42%**

> Merging [#13526](https://codecov.io/gh/pydata/pandas/pull/13526?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **0.08%**

``` diff
@@             master     #13526   diff @@
==========================================
  Files           138        138          
  Lines         51111      51349   +238   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43107      43350   +243   
+ Misses         8004       7999     -5   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [e8e0aae...8b4ad0b](https://codecov.io/gh/pydata/pandas/compare/e8e0aae92d8c93558c06c58817bc9055d9f0a229...8b4ad0b57f4c989bf46ab0c55f26b333adbfb08f)

Tests to perform:
- [x] If pyx files have not changed then cached cython cache **is** reused
- [x] If a pyx file is changed the cython cache **is not** used
- [x] If a pyx file is added the cython cache **is not** used
- [x] If a pyx file is deleted the cython cache **is not** used
- [x] Check a pyx file change when rewritting git history i.e. cache **is not** used

@nparley ok let's try this. ping when green / ready.

There is an interesting gocha with the way travis checks out commits. On a branch travis actually checks out the hash. However, for a PR travis just checks out the latest merge. Therefore if you push lots of changes quickly before travis has started (like I just did early), then builds might not actually reflect the push. (They might all be testing the last push). 

Travis is having problems with Mac OSx runs at the moment. I will not do any more until they fix it.

@jreback OK this is ready to go.

Looks good to me!

thanks @nparley 

let's give this a watch over next few days to make sure its behaving.