BrandonFors/pandas-github-issues huggingface dataset:
TST: Add test for `pd.Timestamp` DST transition
State: closed

- [x] closes #60084(Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Needs Tests
description: Unit test(s) needed to prevent regressions
name: hashing
description: hash_pandas_object
name: Timestamp
description: pd.Timestamp and associated methods

Comments:
Thanks @KevsterAmp
TST (string dtype): clean-up assorted xfails
State: closed

An bunch of assorted xfails that were no longer needed or had a trivial fix

xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Strings
description: String extension data type and string data

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 e7d1964ab7405d54d919bb289318d01e9eb72cd1
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60345: TST (string dtype): clean-up assorted xfails'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60345-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60345 on branch 2.3.x (TST (string dtype): clean-up assorted xfails)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport -> https://github.com/pandas-dev/pandas/pull/60349
BUG: fix to_datetime with np.datetime64[ps] giving wrong conversion
State: closed

- [x] closes #60341 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Nice catch! Can you add a whatsnew note for this to our v3.0.0 documentation?
> Nice catch! Can you add a whatsnew note for this to our v3.0.0 documentation?

Sure. I've added a note.
Awesome work - thanks @yuanx749
CI: update fastparquet xfails for new release
State: closed

Looks like a new release just came out to fix this xfail

Labels:
name: CI
description: Continuous Integration

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 34c080cb91d71a8e900a6c114279e142924c1a64
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60337: CI: update fastparquet xfails for new release'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60337-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60337 on branch 2.3.x (CI: update fastparquet xfails for new release)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport -> https://github.com/pandas-dev/pandas/pull/60344
TST (string dtype): resolve xfails for frame methods
State: closed


Labels:
name: Strings
description: String extension data type and string data
name: Stale

Comments:
Removed the interpolate-related changes in favor of https://github.com/pandas-dev/pandas/pull/60637 (that PR is turning it into a TypeError instead of NotImplementedError, which seems better?)
@jorisvandenbossche @simonjayhawkins any more feedback on this one?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Not actively working on this - let's clear the queue
Added a new parameter to value_counts to keep both counts and normalized values.
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [*] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [*] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [*] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [*] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
Thanks I am also -1 to this enhancement. Closing unless there is an issue opened with positive support for this feature
[backport 2.3.x] TST (string dtype): resolve xfails for frame fillna and replace tests + fix bug in replace for string (#60295)
State: closed

(cherry picked from commit fae3e8034faf66eb8ef00bcbed73d48e4ef791d3)

Backport of https://github.com/pandas-dev/pandas/pull/60295

Labels:

Comments:
This has some failures for which it might be good to first finish https://github.com/pandas-dev/pandas/pull/60183
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Backport PR #60321: TST (string dtype): resolve all xfails in IO pars…
State: closed

Backport of https://github.com/pandas-dev/pandas/pull/60321

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data

Comments:
Hmm not sure how to fix the failing test here. There's a lot of mutability going on in the CSV parser...but I can trace things back to base_parser.py:510 which calls:

```python
index = ensure_index_from_sequences(arrays, names)
```

At that point in time, `arrays` has the right data with the proper dtype:

```
[array(['01'], dtype=object)]
```

but the Index constructor seems to force that to `str`, whereas the test expects `object` to be maintained.

@jorisvandenbossche any thoughts on this one? 
I have a local fix (have to include changes from https://github.com/pandas-dev/pandas/pull/59316 I think), will clean up and push tomorrow
@jorisvandenbossche with your added commit I think this is good to merge (?)
Backport PR #60318 on branch 2.3.x (TST (string dtype): resolve all xfails in JSON IO tests)
State: closed

Backport of https://github.com/pandas-dev/pandas/pull/60318

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO JSON
description: read_json, to_json, json_normalize
name: Strings
description: String extension data type and string data

Comments:
FWIW please link the backport PR and the original PR, and especially remove the "Still Needs Manual Backport" label once you open the backport PR, so we can keep a bit track of what is backported / still needs to be backported

And thanks for the backports!
REF: centralize pyarrow Table to pandas conversions and types_mapper handling
State: closed

We have defined this logic in several places, so defining one helper function to reuse that for the different IO formats.

Labels:
name: Arrow
description: pyarrow functionality

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 12d6f602eea98275553ac456f90201151b1f9bf8
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60324: REF: centralize pyarrow Table to pandas conversions and types_mapper handling'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60324-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60324 on branch 2.3.x (REF: centralize pyarrow Table to pandas conversions and types_mapper handling)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Will backport
Hmm looking at this some more @jorisvandenbossche it looks like the 2.3 branch in parquet.py requires there to be a `split_blocks` argument; I suppose we need to add that to the function here too?
Yeah, either just add that, or add a generic `to_pandas_kwargs` that get passed through, I would say
Backport PR https://github.com/pandas-dev/pandas/pull/60332
Add warning for sharex/sharey when subplots=False in plot method
State: closed

- [x] closes #60299
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Closing as the issue needs more discussion first
TST (string dtype): resolve all xfails in IO parser tests
State: closed

There are two remaining xfails left: one related to invalid unicode (that errors if using the pyarrow-backed string dtype, so we should probably have a fall back to object dtype fo that case), and another one about specifying the `names` keyword with the pyarrow engine giving object-dtype columns.


xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: IO CSV
description: read_csv, to_csv
name: Strings
description: String extension data type and string data

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 ee3c18f51b393893ed6e31214c7be2f9427ce0c9
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60321: TST (string dtype): resolve all xfails in IO parser tests'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60321-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60321 on branch 2.3.x (TST (string dtype): resolve all xfails in IO parser tests)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Will backport this one too
Manual backport -> https://github.com/pandas-dev/pandas/pull/60330
TST (string dtype): resolve all xfails in JSON IO tests
State: closed


Labels:
name: IO JSON
description: read_json, to_json, json_normalize
name: Strings
description: String extension data type and string data

Comments:
While looking at centralizing `to_pandas()` conversion, I noticed that for JSON we are not yet handling the case of the string dtype for the pyarrow engine (engine, not dtype_backend), so it's a bit surprising actually that all tests are passing here ...
Interesting - I didn't even realize we had a pyarrow engine for JSON!

Is your expectation that those cases would be returning `dtype=object` and cause a failure?
> Is your expectation that those cases would be returning `dtype=object` and cause a failure?

Yes, at least if we have proper testing of it ..
But let's do that in a next PR, going to merge this one already
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 9bc88c79e6fd146a44970309bacc90490fdec590
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60318: TST (string dtype): resolve all xfails in JSON IO tests'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60318-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60318 on branch 2.3.x (TST (string dtype): resolve all xfails in JSON IO tests)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Alright - I'll take a look. I wonder if there's some post-processing in JSON that is masking that...

I'll take care of the backport as well. Thanks for merging
Manual backport -> https://github.com/pandas-dev/pandas/pull/60327
[backport 2.3.x] String dtype: enable in SQL IO + resolve all xfails (#60255)
State: closed

(cherry picked from commit ba4d1cfdda14bf521ff91d6ad432b21095c417fd)

Backport of https://github.com/pandas-dev/pandas/pull/60255

Labels:

Comments:
Thanks for opening. I'll take a look at the failures
As far as I can tell this issue might exist at a deeper level on the backport branch. On main, I see this behavior:

```python
>>> import pandas as pd
>>> import pandas._libs.lib as lib
>>> import numpy as np

>>> arr = np.array(["a", "b", None])
>>> lib.maybe_convert_objects(arr, convert_to_nullable_dtype=True, convert_non_numeric=True)
<StringArray>
['a', 'b', <NA>]
Length: 3, dtype: string
```

that same call on the backport branch yields np.nan as the missing value sentinel:

```python
>>> lib.maybe_convert_objects(arr, convert_to_nullable_dtype=True, convert_non_numeric=True)
<StringArrayNumpySemantics>
['a', 'b', nan]
Length: 3, dtype: str
```

that is unexpected right?
I think the backport branch was missing the change in https://github.com/pandas-dev/pandas/pull/59487 shown in the second commit here; adding that back in locally gets the tests to pass

Not sure where that got removed, but assuming during the setup of the backport branch accidentally
Thanks for figuring that out!

There is still something related to the datetime64 resolution, will take a look at that
I think the datetime issues were caused by a very subtle backport issue. Hopefully resolved by latest commit
Is there a way to restart the failed pre-commit job? I think that is spurious
Thanks for the fix!

> Is there a way to restart the failed pre-commit job? I think that is spurious

No idea. I typically just ignore it if it times out
TST (string dtype): resolve all easy xfails in pandas/tests/groupby
State: closed

There are a few remaining xfails that are harder to fix (require some investigation or probably an actual code fix, added some comments where I have a clue for the reason), but for the rest this resolves most xfails in `pandas/tests/groupby`

xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Strings
description: String extension data type and string data

Comments:
Thanks @jorisvandenbossche 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 c4a20261c337d68dc470fb6fd6a5505e2c7348d0
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60314: TST (string dtype): resolve all easy xfails in pandas/tests/groupby'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60314-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60314 on branch 2.3.x (TST (string dtype): resolve all easy xfails in pandas/tests/groupby)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport -> https://github.com/pandas-dev/pandas/pull/60317
BUG: Supress  `read_json` Spurious `to_datetime` FutureWarning
State: closed

- [x] closes #59511 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
https://github.com/pandas-dev/pandas/issues/59511
I assume CI failures are unrelated, though they are on the same `read_json` tests. The tests still seems to fail even if I undo all my changes in local
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
TST (string dtype): resolve xfails in pandas/tests/apply + raise TypeError for ArrowArray accumulate
State: closed

Resolve all remaining string xfails in `pandas/tests/apply`, except for one about cumsum not yet implemented for string dtype.

Also ensures we don't raise a pyarrow specific error in case of the pyarrow-backend for accumulation methods.

xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Strings
description: String extension data type and string data

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 fba5f08f048215a6e0a578f8bad7b7f2c9ee8eef
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60312: TST (string dtype): resolve xfails in pandas/tests/apply + raise TypeError for ArrowArray accumulate'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60312-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60312 on branch 2.3.x (TST (string dtype): resolve xfails in pandas/tests/apply + raise TypeError for ArrowArray accumulate)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Since you've got a ton going on I'm happy to take care of this backport
Manual backport -> https://github.com/pandas-dev/pandas/pull/60328
BUG: Disabling pandas option `display.html.use_mathjax` has no effect; add `mathjax_ignore` to fix
State: closed

- [x] closes #59884 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Styler
description: conditional formatting using DataFrame.style

Comments:
Thanks @KevsterAmp
DOC: Update from_records docstring to not mention DataFrame as valid input
State: closed



- closes #60307 

Removed mention of DataFrame as an acceptable input type, as it is no longer supported
`
        Convert structured or record ndarray to DataFrame.

        Creates a DataFrame object from a structured ndarray, sequence of
        tuples or dicts.

`

Ensured compatibility with numpydoc format by validating against validate_docstrings.py

Labels:
name: Docs

Comments:
Thanks @U-S-jun