Refactor CI
State: closed

This PR 1 from this list https://github.com/pandas-dev/pandas/pull/46346#issuecomment-1076736180

cc @mroeschke 

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration

Comments:
Looks like only 1 check ran
GHA has serious problems right now
Can we hold off on removing Azure pipelines for now? Right now, jobs queue for very long times >2 hours(sometimes up to 4 hours) when lots of PRs are merged. I'm trying to clear out some redundant builds right now, but that will probably take me a while before I get the builds down to a reasonable number(~20 jobs per build). 
Agreed that this PR seems to tackle too much still for a thorough review, especially for CI changes where we need to be careful about things passing. Generally it's easier if 1 PR tackles 1 general issue/theme
> Right now, jobs queue for very long times >2 hours(sometimes up to 4 hours) when lots of PRs are merged

Is there any way to change to a large GHA plan? Maybe through some GH open source sponsoring program or some Pandas sponsor?


Re even more splitting, would it be a viable option to ship most of the shared code in `.github/actions` but only a single (or so) changed workflow? That would reduce the size of the changes substantially but the PR might include changes not relevant to the changed workflows (because only relevant to changes in workflows in a subsequent PR).
> > Right now, jobs queue for very long times >2 hours(sometimes up to 4 hours) when lots of PRs are merged
> 
> Is there any way to change to a large GHA plan? Maybe through some GH open source sponsoring program or some Pandas sponsor?

Also maybe we can bail out earlier if there are test failures. I thought that maybe w e could come up with a very small test of "pre tests" that we can run in eg. 60 s and that would exit the test run if failed.
> Is there any way to change to a large GHA plan?

We could probably reach out to Github to see what's possible. Probably good to bring up at the next developer meeting. FWIW I don't think the long queue times are a huge deal. Minus the redundancy of testing which wastes compute resources, development is fairly asynchronous over longer time spans (especially reviewer availability) without major deadlines so the waiting is not major IMO.

> Re even more splitting

Some of the changes I see that could be a single PR

* `pytest-asyncio>=0.17`
* Bumping `actions/checkout`
* `bash -el`
* Reordering the yaml of the `include` jobs
* Refactoring sdist workflow
* setup actions
* build action
* test action
> Probably good to bring up at the next developer meeting.

Are you suggesting that I should join the meeting (where can I find info?) or are you going to bring it up?

> FWIW I don't think the long queue times are a huge deal.

I think they are because they create a very long feedback loop for contributors. It's fine if tests take a long time but I think it's worth investing some effort into reducing the latency between a push and the first test being executed.

> Some of the changes I see that could be a single PR

Working on a new PR!
@jonashaag the CI is very large on pandas and we are already using a large plan. it is unlikey we are going to be able to change anything here. The cycle times are what they are. 
> > > Right now, jobs queue for very long times >2 hours(sometimes up to 4 hours) when lots of PRs are merged
> > 
> > 
> > Is there any way to change to a large GHA plan? Maybe through some GH open source sponsoring program or some Pandas sponsor?
> 
> Also maybe we can bail out earlier if there are test failures. I thought that maybe w e could come up with a very small test of "pre tests" that we can run in eg. 60 s and that would exit the test run if failed.

Going along a similar vein, maybe we can try making the slow CI depend on non slow CI and see if that helps(we can cache the built pandas from the non-slow CI, and install from that for the slow CI).
What's really curious is that the GHA machines are SO incredibly slow. On my M1 machine with no pytest-xdist the full test suite takes 8 minutes, while on GHA with 2 cores it takes ~ 45 minutes.