BrandonFors/pandas-github-issues huggingface dataset:
Backport PR #46567 on branch 1.4.x (BUG: groupby().rolling(freq) with monotonic dates within groups #46065 )
State: closed

Backport PRs #46567, #46087

@mroeschke marked as draft as this PR also backports #46087 (which conflicts that caused the autobackport to fail)

The only conflict doing this was the release note in 1.5 (which would need a separate PR to move on main) (side note: why was the release note for #46567 in bug fix section and not regression?)

#46087 is labelled as an enhancement but could also be considered a bugfix? Although saying that, I think we generally don't consider changes to error message to be API changes (only changes to exception type)?

wdyt about backporting #46087? No pressure, but if you prefer not to, I will leave it to you to do the manual backport ! (you are more familiar with this code so less likely to make a mistake)

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Window
description: rolling, ewma, expanding

Comments:
> why was the release note for https://github.com/pandas-dev/pandas/pull/46567 in bug fix section and not regression?

Ah sorry that was an oversight on my end. It should be in the regression section

> wdyt about backporting https://github.com/pandas-dev/pandas/pull/46087?

Sure, I'm ok with backporting since in essence it just changes the error message

> > why was the release note for #46567 in bug fix section and not regression?
> 
> Ah sorry that was an oversight on my end. It should be in the regression section

no problem. will move it later (once the other note is synced up with main)
DOC: Fix errors detected by sphinx-lint
State: closed

Found a few issues on the doc with [sphinx-lint](https://github.com/sphinx-contrib/sphinx-lint).

Fixed them.

Labels:
name: Docs
name: Clean

Comments:
Nice, thanks - could you add this to https://github.com/pandas-dev/pandas/blob/672798ec7b5565366be24aa4f57f6374f3a5e0fd/.pre-commit-config.yaml
I don't understand the error :(
looks unrelated, don't worry about it
Should I merge main to get it fixed (in other words, do I have something to do?)
I re-ran and it passed - all good, thanks!
REGR: groupby.transform producing segfault
State: closed

- [x] closes #46566 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

No whatsnew because this is only on main currently.

Labels:
name: Bug
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @rhshadrach
REF: expose pandas_timedelta_to_timedeltastruct
State: closed

Broken off from a branch implementing non-nano Timedelta scalar.

`cmp_dtstructs` will end up being used by both `Timedelta.__richcmp__` and `Timestamp.__richcmo__`

Labels:
name: Refactor
description: Internal refactoring of code
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @jbrockmendel
POC For docker compose
State: closed

- [X] closes #46532 

POC using docker compose, which is also used by the arrow project. 

The idea here is that we can simply run `docker compose build db-testing` to build an image with postgres (can later add mysql) and our minimal development requirements then `docker compose run --rm db-testing` to run relevant tests. This can be done both by a developer as well as on GH actions.

This still needs a bit more work as it currently muddles user permissions on the host when building pandas

cc @jonashaag who looks to have been doing some awesome work on CI lately

Labels:
name: CI
description: Continuous Integration

Comments:
I am in favor of having what users tests locally use the same setup as what the CI runs which containers help enforce.

We should keep in mind that Docker is a freemium tool (limited Dockerhub pulls over a period of time albeit maybe hard to hit & some free container registries out there).

I'll be interested to see how we can leverage docker-compose with the CI providers - initially my gut feels it might just be a lateral move - and maintain testing coverage across different platform versions. We're almost at the point where we can say

```
for platform in [windows, mac, ubuntu]:
    for python_version in [3.8, 3.9, 3.10]:
        test with platform + python_version + all required & optional dependencies
```

I would suggest the docker-compose POC mirrors the jobs that GHA currently runs for an easier comparison to our current CI setup. So in terms of testing, all tests can be run for a particular platform + Python version. 
Awesome feedback. The one area where docker is definitely an upgrade over current CI is reproducibility as an end user. GHA are good but AFAIK not something you can reproduce locally. I think the DB tests in particular are an area where we struggle with that

In the future there might also be a use case where we create base images for minimum pinned versions and always reference them, rather than building from scratch in GHA every time. This could help reduce build / test times a bit

We already have this with the existing Dockerfile, but if we roll that into compose we can have a consistent way to create a DEV environment (especially for new users) that our current CI can't help with
> I am in favor of having what users tests locally use the same setup as what the CI runs which containers help enforce.

+1
So this works now with a simple `docker compose up` - starts both DB services and runs all the tests

Punting for now. Can reopen if it becomes more clearly useful
TYP: setter for index/columns property-like (AxisProperty)
State: closed


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Don't run GHA CI for Azure changes and vice-versa
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration

Comments:
i don't like changes like this as it makes much harder to understand what is running
removing milestone for now.
We don't need this once we switch away from Azure
Backport PR #46558: CI: pre-commit autoupdate to fix CI
State: closed

Backport PR #46558

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
@MarcoGorelli There are a few CI PRs to manually backport. starting with #46543 (have opened #46559)

It maybe better to not merge this one yet. (makes autobackports more likely to succeed)

Out of curiosity, where were the conflicts that prevented the autobackport of #46558 succeeding
sure, will hold off on this one

they were conflicts with another PR which updated `black` version pins
Thanks. Have marked as draft and will do the backports in order, so this one after #46541 and #46540
> they were conflicts with another PR which updated `black` version pins

I guess #46490 which we didn't backport. so probably autobackport won't suceed even after getting all the ci stuff in sync.
code check are failing, I assume since we did not backport #45752

in https://github.com/pandas-dev/pandas/pull/46558#issuecomment-1081854269, I said ...

> I assume will will get the same error on 1.4.x?

even though we are not seeing the same failures on 1.4.x

It maybe that since black on 1.4.x is pinned to 21.12b0 whereas on main it was 22.1.0 that we don't need the backport.

Alternatively, since the black changes should not affect code behavior, we could run the latest black as a commit here (I assume that would be easier than backporting #45752 and #46490)
Looks like it works fine in 1.4.x - maybe `21.12b0` isn't affected by the latest `click` release then?

I'd suggest to then either not backport `black` changes at all, or if `click` does cause issues, then just to pin that in `additional_requirements` for this hook

Or will not backporting `black` changes make other backports tricker?
> Or will not backporting `black` changes make other backports tricker?

There we many files changed in #45752 so there is perhaps a greater risk that a manual backport is required in the future due to a conflict.

I'm +1 on updating the code on 1.4.x to the latest black (although IIRC we have not tended to do this in the past)
sure, I've done this in https://github.com/pandas-dev/pandas/pull/46576, will close this one
tocsv_interval _categories
State: closed

- [x] closes #46297
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO CSV
description: read_csv, to_csv
name: Categorical
description: Categorical Data Type

Comments:
Thanks @Kyrpel for the PR. does this also fix the related issue #46812?

can you add a release note doc/source/whatsnew/v1.4.3.rst
Thanks @Kyrpel for the PR.

This fix is currently adding some more code where the regression was caused by a code addition in #44930.

https://github.com/pandas-dev/pandas/blob/009c4c622c739941ae3dfde4bc4117eb25172500/pandas/core/internals/blocks.py#L2265-L2271

whereas before #44930 the conversion was done in ...

https://github.com/pandas-dev/pandas/blob/009c4c622c739941ae3dfde4bc4117eb25172500/pandas/core/internals/blocks.py#L2318-L2323

Now removing the code addition in #44930 also fixes this issue.

Now since the `astype(object)` for categorical containing Timestamps was also fixed in #18024, i'm not sure why the special casing for Categorical was added to `to_native_types`.

Going back to this issue. `repr` already still works and according to some code comments, `to_csv`/`repr` are supposed to be sharing the same `to_native_types` code.

we also have the related issue https://github.com/pandas-dev/pandas/issues/46812, and I may need to update this observation once I have looked at that one. I suspect that both issues should probably be fixed togther.
> Now since the `astype(object)` for categorical containing Timestamps was also fixed in #18024, i'm not sure why the special casing for Categorical was added to `to_native_types`.

should have been containing Timestamps was also fixed in #18024 -> containing Timestamps #18024 was also fixed in #44930

cc @jbrockmendel @phofl (participants in https://github.com/pandas-dev/pandas/pull/44930)
Thanks @Kyrpel for the PR. closing as superseded by #47347.
Backport of CI related PRs on 1.4.x
State: closed

backports #46543, #46401, #46347, #46260, #46426, #46016, #46370 and #46358

(will change milestones on those, once CI here is successful and merged.)

Labels:
name: CI
description: Continuous Integration

Comments:
ooh. looks like the py38_32bit pipeline is fixed by one of these PRs too.
thanks @simonjayhawkins
CI: pre-commit autoupdate to fix CI
State: closed

new click release broke previous version of `black`, see https://github.com/psf/black/issues/2964

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
I assume will will get the same error on 1.4.x?
> I assume will will get the same error on 1.4.x?

hmm, merged #46550 a couple of hours ago after the problem was noticed on main. pre-commit was green on 1.4.x. more investigation required.
I think it's just that on the 1.4.x branch the run used the old cached dependencies:
```
Cache Size: ~63 MB (65887285 B)
```
which would've had the previous `click` version

Don't know how long the cache lasts, but I think this should probably be backported
> OK to merge once Code Checks / pre-commit is green 

merging then
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 8f442441e02f7239fd7d34c2e177a39125062e05
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #46558: CI: pre-commit autoupdate to fix CI'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-46558-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #46558 on branch 1.4.x (CI: pre-commit autoupdate to fix CI)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@meeseeksdev backport 1.4.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 8f442441e02f7239fd7d34c2e177a39125062e05
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #46558: CI: pre-commit autoupdate to fix CI'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-46558-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #46558 on branch 1.4.x (CI: pre-commit autoupdate to fix CI)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
Backport PR #46358 on branch 1.4.x (CI: Debug Windows stack overflow)
State: closed

Backport PR #46358: CI: Debug Windows stack overflow

Labels:
name: CI
description: Continuous Integration

Comments:
marked as draft as will probably create a single PR with several CI related backports otherwise will take ages with CI cycle waiting for each one until can autobackport the next.
closed in favor of #46559
Close FastParquet file even on error
State: closed

- [x] closes #46555
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO Parquet
description: parquet, feather

Comments:
> going to need a whatsnew note in I/O for 1.5

done!
Hello @flying-sheep! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-03-30 15:25:57 UTC
thanks @flying-sheep
Speed up a test
State: closed

Saves 30s of test time, it ain't much but it's honest work :-) Also moves a magic number to a constant.

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO CSV
description: read_csv, to_csv

Comments:
thanks @jonashaag all for faster tests, but we also cannot change them like this (which doesn't really test much). I basically am always -1 on monkey patching as it 99% of the time doesnt' catch anything.
Actually my version of the test fails before the #8621 fix, I double checked. So it actually tests the bug, just with a smaller number
BUG: pd.concat with identical key leads to multi-indexing error
State: closed

- [x] closes #46519 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex

Comments:
Hello @GYHHAHA! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-04-04 20:33:50 UTC
Thanks @GYHHAHA
Pin pytest-asyncio>=0.17
State: closed

Required for "asyncio_mode" setting that was added to pyproject.toml.

Part of https://github.com/pandas-dev/pandas/pull/46493#issuecomment-1079541210

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration

Comments:
thanks @jonashaag ok for backport as this is a CI / dev env only change.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 a93792d7db632fa05225046f77e2ab6994c0d913
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #46543: Pin pytest-asyncio>=0.17'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-46543-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #46543 on branch 1.4.x (Pin pytest-asyncio>=0.17)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@meeseeksdev backport 1.4.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 a93792d7db632fa05225046f77e2ab6994c0d913
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #46543: Pin pytest-asyncio>=0.17'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-46543-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #46543 on branch 1.4.x (Pin pytest-asyncio>=0.17)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
Reorder YAML
State: closed

Part of https://github.com/pandas-dev/pandas/pull/46493#issuecomment-1079541210

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration

Comments:
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.