CI/TST: update pyarrow pin in windows-3.8 env / fix failing parquet test on windows
State: closed

Closes #45344

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
So updating pyarrow from 1.0 to 2.0 seems to highlight some other bugs ..

(edit: the xfails for this have been updated in https://github.com/pandas-dev/pandas/pull/45257 in the meantime)
> So updating pyarrow from 1.0 to 2.0 seems to highlight some other bugs ..

Something like this seems to happen frequently.  I've got an under-utilized ubuntu machine that i could set up with 'tox' to run the tests daily(ish) with a thorough(ish) matrix of pyarrow versions. Would that be useful?  If this is windows-specific then never mind.
closing in favor of #45354 which is already merged & backported
> closing in favor of #45354 which is already merged & backported

#45354 is only xfailing the test, while this PR is an initial attempt to actually fix #45344 (which is still open)

> Something like this seems to happen frequently. I've got an under-utilized ubuntu machine that i could set up with 'tox' to run the tests daily(ish) with a thorough(ish) matrix of pyarrow versions. Would that be useful? If this is windows-specific then never mind.

I don't think it is windows-specific (it's a fsspec / pyarrow incompatible version combination). But I also don't think it's necessarily needed for you to set up separate testing against more versions of pyarrow (or at least it _should_ not be needed, we should "simply" do this better on our CI). I would first try to diversify the version in our own CI matrix (eg we can already reduce the number of builds that pins against pyarrow=1.0.1). 
this looks good thanks @jorisvandenbossche
@meeseeksdev backport 1.4.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -m1 91f12c44d94b4d399e3f2321edb7c35c91998242
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #45364: CI/TST: update pyarrow pin in windows-3.8 env / fix failing parquet test on windows'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-45364-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #45364 on branch 1.4.x (CI/TST: update pyarrow pin in windows-3.8 env / fix failing parquet test on windows)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -m1 91f12c44d94b4d399e3f2321edb7c35c91998242
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #45364: CI/TST: update pyarrow pin in windows-3.8 env / fix failing parquet test on windows'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-45364-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #45364 on branch 1.4.x (CI/TST: update pyarrow pin in windows-3.8 env / fix failing parquet test on windows)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@jorisvandenbossche if you wouldn't mind pushing the manual backport here
-> https://github.com/pandas-dev/pandas/pull/45450