CI: Test pyarrow nightly instead of intermediate versions
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration
name: Arrow
description: pyarrow functionality

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 4a2c06c8a5e4b12f7850b834eb10f1fa1f302f92
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #52211: CI: Test pyarrow nightly instead of intermediate versions'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-52211-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #52211 on branch 2.0.x (CI: Test pyarrow nightly instead of intermediate versions)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Do we have an issue to add back testing with multiple versions of pyarrow? I understand this PR reduced the CI load, but now we have several supported versions of pyarrow that we are not testing at all. 

And it actually seems that in practice, something else in the environment is causing a constraint on older pyarrow, so all our standard builds are using pyarrow 9.0.0. Which means we are not even testing the latest released pyarrow version at the moment?

We have standard env files for python 3.9, 3.10 and 3.11 at the moment, we could easily test three different pyarrow versions in those? (xref https://github.com/pandas-dev/pandas/pull/51308#issuecomment-1426406042)
Testing our actions-311.yaml env file locally, and then trying to upgrade pyarrow causes a downgrade of pandas-gbq, so the bump of the minimum version of this might have caused to test with an older version of pyarrow.
We don't have an open issue to discuss adding back testing with multiple version, but it would be good to have one to discuss. Yeah maybe we could have jobs that tests the required dependencies + pyarrow 8, 9, 10, ...
> Do we have an issue to add back testing with multiple versions of pyarrow? I understand this PR reduced the CI load, but now we have several supported versions of pyarrow that we are not testing at all.

Are there issues with pyarrow versions in the middle of our support range breaking?

FWIW, for numpy we only test the min, the latest, and numpy dev, and it seems to work fine.

Although the build matrix has been reduced by fusing jobs together, the individual jobs themselves take longer now (~1hr per job), so I'm still a little worried about the runners clogging up (We support a wide range of pyarrow rn, which is pyarrow 7-12 I think). 


> And it actually seems that in practice, something else in the environment is causing a constraint on older pyarrow, so all our standard builds are using pyarrow 9.0.0. Which means we are not even testing the latest released pyarrow version at the moment?

This seems like an issue. 

We could try adding a "clean" job without any dependencies, that would have pyarrow in it (this depends on PDEP-10 being accepted).

In this case though, we are probably just pulling a very old version of pandas-gbq (mamba bug?).