CI: split Windows Azure tests in half
State: closed

Trying an experiment for Windows CI where `PYTEST_WORKERS` is `auto` rather than 2.


Labels:
name: CI
description: Continuous Integration
name: Windows
description: Windows OS

Comments:
how'd this work out?
> how'd this work out?

`auto` doesn't solve the problem.  Current hypothesis is that the expression for skipping tests is wrong.  Here's the evidence:

For the Windows py38 tests, which are "not slow and not network", we get: `122956 passed, 6965 skipped`
For the Windows py39 tests, which are ""not slow and not network and not high_memory"", we get `124788 passed, 4943 skipped`

Doesn't make sense that skipping _more_ tests in the expression would yield less tests skipped.

That test is running now....

The other issue is that it isn't clear to me how many cores are actually assigned to the Azure instances.  Using `auto` means use as many cores as are available.  In the first set of tests under this PR, py38 and py39 both used two cores.  In the latest push, py38 is using 4 cores and py39 is using 2.  The more cores we can get, the faster these tests will run.


38 and 39 might have different deps though
> In the latest push, py38 is using 4 cores and py39 is using 2. The more cores we can get, the faster these tests will run.

Just noting from looking at this in #42236, seemed like the opposite was actually true based on looking at a few runs using different numbers of cores. 

Revised hypothesis.  The number of cores that we get from Azure is variable.  But the issue here is that if you only have 2 cores and use both of them, performance can get _worse_ than if you use just 1 core.  Here is some evidence.  On my 16 core laptop, I ran `pytest --skip-slow -m "not single" -n XX pandas/tests/groupby`  for XX=1,2,4,8,16 and auto.  Here are the timing results:

| XX | Time (secs) |
| ------ | ------- |
| 16  | 32.09 | 
|  8  |  30.16 | 
|  4 |   37.56 | 
|  2  |  55.04 | 
|  1  | 90.53 | 
| auto |  28.87 |

For "auto", it created 8 processes.

Note that using all 16 cores doesn't help - that's because these are hyperthreaded cores, and using them usually doesn't help computational performance.   Secondly, even going from 1 core to 8 only sped things up by a factor of 3.

So if Azure gives us 2 cores, and we use both, that could be worse than just using 1 core.  My latest commit is doing that, and also (hopefully) printing out the CPU info.




oh so this really *could* be a timeout?

also is there a way to request 4 cores (min) from azure for these builds?
> oh so this really _could_ be a timeout?

Yes.  The last test confirms it.  Here is the processor info that was obtained (which was the same for the `py38` and `py39` runs):
```
Caption                            DeviceID  MaxClockSpeed  Name                                            NumberOfCores  

Intel64 Family 6 Model 85 Stepping 7  CPU0      2594           Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz  2              
```
In the last test, I had it not do any parallelization, and the `py38` run timed out, the `py39` run did not.  So I think this has to do with where the virtual container is sitting on the hardware.  Most of the time before, I'd see `py38` succeed and `py39` time out.

> 
> also is there a way to request 4 cores (min) from azure for these builds?

I don't know the answer to that!   Who is responsible for the relationship we have with azure?


> I don't know the answer to that! Who is responsible for the relationship we have with azure?

umm, don't really have anything specific
Another possible solution is to split the tests for Azure.  Here's an example from Microsoft on how to do that:
https://github.com/PBoraMSFT/ParallelTestingSample-Python
maybe best is just to remove some tests (e.g. have a marker that we dont' use here) and/or move some to a slow build.
> maybe best is just to remove some tests (e.g. have a marker that we dont' use here) and/or move some to a slow build.

Issue here is that the slowest test is 10 seconds.  Our issue is volume (over 124K tests) and on Azure, we're running on a "slow" machine compared to Travis.


> > maybe best is just to remove some tests (e.g. have a marker that we dont' use here) and/or move some to a slow build.
> 
> Issue here is that the slowest test is 10 seconds. Our issue is volume (over 124K tests) and on Azure, we're running on a "slow" machine compared to Travis.

right one way is for example to NOT run *any* pandas/tests/windows (which @mroeschke disabled for a while)
not great, but we do really test these
If we cant get more cores/build, could we run more azure builds, e.g. for each of the current azure builds split it in two with one ignoring tests/window and the other doing only tests/window?
Or maybe we can test a separate Github Actions Window runner for a subset of tests (seems to only be a 2 core CPU https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources)
I'm trying something else here by setting up a `PYTEST_TARGET` in the azure pipelines and then for Windows having separate jobs where I separate the tests into ` "pandas/tests/[a-i]*"` and `"pandas/tests/[j-z]*"`
looking good

can u close and reopen to force a run again (just checking on the timeouts)
> can u close and reopen to force a run again (just checking on the timeouts)

Closing and reopening..
@Dr-Irv this looks good. i think splitting the macos tests along the same lines makes sense as well (in this PR or followon)
@simonjayhawkins lmk if you want to backport (either way ok by me)
@jreback all green

We also have the option of speeding up some of the other checks by doing a similar split of the tests.  Some of them are taking close to an hour.   That would allow people to get quicker feedback on their PRs.  If you think that's a good idea, I could do another PR for that.
> @jreback all green
> 
> We also have the option of speeding up some of the other checks by doing a similar split of the tests. Some of them are taking close to an hour. That would allow people to get quicker feedback on their PRs. If you think that's a good idea, I could do another PR for that.

yep, we actually have a pretty large azure allocation, so a *few* more jobs per PR is fine.
going to backport this (@simonjayhawkins can always not merge it on 1.3 if too many conflicts)
thanks @Dr-Irv 
@meeseeksdev backport 1.3.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.3.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 6e19bdc5ac2387525a850e880b8f67f7958cf6d1
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #43468: CI: split Windows Azure tests in half'
```

4. Push to a named branch :

```
git push YOURFORK 1.3.x:auto-backport-of-pr-43468-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #43468 on branch 1.3.x (CI: split Windows Azure tests in half)"

And apply the correct labels and milestones.

Congratulation you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove `Still Needs Manual Backport` label once the PR gets merged.

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.3.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 6e19bdc5ac2387525a850e880b8f67f7958cf6d1
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #43468: CI: split Windows Azure tests in half'
```

4. Push to a named branch :

```
git push YOURFORK 1.3.x:auto-backport-of-pr-43468-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #43468 on branch 1.3.x (CI: split Windows Azure tests in half)"

And apply the correct labels and milestones.

Congratulation you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove `Still Needs Manual Backport` label once the PR gets merged.

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@Dr-Irv if you can push the backport as above (guess we have a conflict)
> @Dr-Irv if you can push the backport as above (guess we have a conflict)

Created this PR https://github.com/pandas-dev/pandas/pull/43496

Not sure which labels and milestones to put on it, and the backport PR is showing conflicts

@jreback   Bad PR #43496 (was merging to master).  New PR #43497 merges to 1.3.x .
Awesome, thanks for figuring this out @Dr-Irv