BrandonFors/pandas-github-issues huggingface dataset:
Implement Buffer Protocol for PandasBuffer
State: closed

See https://github.com/pandas-dev/pandas/pull/55227#discussion_r1370815016

Labels:
name: Stale
name: Interchange
description: Dataframe Interchange Protocol

Comments:
Need to investigate the validity buffer in more detail. Arrow would use bits for that so some of the sizing may be off, if that hasn't already been converted to bytes elsewhere in the interchange 
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Closing for now - may revisit later
REGR: merge_asof raising TypeError for "by" with datelike dtypes
State: closed

- [x] closes #55453 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.2.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Thanks @lukemanley
DOC: Update release date for 2.1.2
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @lithomas1
BUG: shape issue when concatenating different datetime resolution (#55374)
State: closed

- [ ] closes #55374 & #55067
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests)
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/v2.1.2.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
#53641 is the correct approach for this
Thanks for the PR, but since it appears https://github.com/pandas-dev/pandas/pull/53641 is already addressing this issue closing in favor of that issue
BUG: DTA/TDA constructors with mismatched values/dtype resolutions
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
Thanks @jbrockmendel
BUG: infer_string not inferring string dtype when NA is first value
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality

Comments:
thanks @phofl.
CI: Fix pyreadstat upgrade change
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
Thanks @phofl 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.1.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 0de3bcbcb5bf3010173d478f4c327e4defcbe6b2
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #55653: CI: Fix pyreadstat upgrade change'
```

4. Push to a named branch:

```
git push YOURFORK 2.1.x:auto-backport-of-pr-55653-on-2.1.x
```

5. Create a PR against branch 2.1.x, I would have named this PR:

> "Backport PR #55653 on branch 2.1.x (CI: Fix pyreadstat upgrade change)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
IIRC this is a 2.2 feature so this may not need a backport

even better
CLN: remove unused in tools.datetimes
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Clean

Comments:
Thanks @jbrockmendel
Add missing space in read_csv usecols out-of-bounds case
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ X ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Thanks @mlbileschi
Fixed GIL regressions with Cython 3
State: closed

ref https://github.com/pandas-dev/pandas/pull/55179

The main issue was that the functions returning `double` were not marked `noexcept`, so in Cython 3.0 these now explicitly check the global Python error indicator, which requires a reacquisition of the GIL

The changes to `to_fw_string` were to fix warnings generated by Cython that subsequently appeared:

```
 [2/4] Compiling Cython source /home/willayd/clones/pandas/pandas/_libs/parsers.pyx
  performance hint: /home/willayd/clones/pandas/pandas/_libs/parsers.pyx:1622:5: Exception check on '_to_fw_string_nogil' will always require the GIL to be acquired.
  Possible solutions:
        1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
        2. Use an 'int' return type on the function to allow an error code to be returned.
  performance hint: /home/willayd/clones/pandas/pandas/_libs/parsers.pyx:1617:27: Exception check will always require the GIL to be acquired.
  Possible solutions:
        1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
        2. Use an 'int' return type on the function to allow an error code to be returned.
  performance hint: /home/willayd/clones/pandas/pandas/_libs/parsers.pyx:1707:42: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: /home/willayd/clones/pandas/pandas/_libs/parsers.pyx:1731:38: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
```

We maybe shouldn't just be swallowing errors like this, but that is a different problem for a different day...for now this should get us back to 0.29 performance


Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Which functions are you looking at? In theory I think we only want to do this for non-integral return types. The integral returning functions should be checking the error sentinel first before trying to acquire the GIL, which I think is more reasonable than `noexcept`
Thanks @WillAyd 
> Which functions are you looking at? In theory I think we only want to do this for non-integral return types. _The integral returning functions should be checking the error sentinel first before trying to acquire the GIL_, which I think is more reasonable than `noexcept`

They were indeed all integral return types. Can you explain further the italic bit above? Is the "should be checking" bit done by the caller? Isn't whether an exception was raised checked before we get back to the caller?
ENH: read_stata return non-nano
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Pre-emptively addresses issues that come up when implementing #55564

cc @bashtage

Labels:
name: IO Stata
description: read_stata, to_stata
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Does the writer code automatically handle non-ns timestamps? Or does it need some work?  I give some time if needed.

> Does the writer code automatically handle non-ns timestamps? Or does it need some work? I give some time if needed.

IIUC _datetime_to_stata_elapsed_vec handles this correctly, but another pair of eyeballs would be a good idea
@bashtage gentle ping
Will take a look later today.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks @jbrockmendel
DOC: Move whatsnew in 2.1.2
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.



Labels:
name: Docs

Comments:
Thanks @phofl
BUG: setitem casting object Index to arrow strings
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @jbrockmendel 

I think we can treat this as a bug for the new option not a case for deprecation?

I'll open a deprecation PR for all other scenarios after this is in

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality

Comments:
cc @mroeschke 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.1.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 639bd6695940be8737db5c47b71b7ec6c4c88fb7
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #55639: BUG: setitem casting object Index to arrow strings'
```

4. Push to a named branch:

```
git push YOURFORK 2.1.x:auto-backport-of-pr-55639-on-2.1.x
```

5. Create a PR against branch 2.1.x, I would have named this PR:

> "Backport PR #55639 on branch 2.1.x (BUG: setitem casting object Index to arrow strings)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BUG: indexing raises for ea into numpy series
State: closed

- [x] closes #55604 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Thanks @phofl 
Has this been released? I still encounter error that's similar to [55604](https://github.com/pandas-dev/pandas/issues/55604)
Not yet. Hopefully in January this should be released
fix #55489
State: closed

- [BUG ] closes #55489
- Added _**"This will necessarily result in ``object`` dtype."**_ to the pandas.io.read_excel() docstring


Labels:

Comments:
@ziadk - you do not need to update your PR branch each time the main branch is updated. Doing so kicks off all the tests. Periodically (every few days or so) is okay, and especially when there are merge conflicts.
@ziadk - this looked pretty good, likely just needed to fix the line being too long (we don't allow more than 88 characters on a line) and then add the missing space between the period and the next sentence. Are you interested in continuing?
BUG: value_counts returning incorrect dtype for string dtype
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Groupby
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Strings
description: String extension data type and string data

Comments:
added a test
Thanks @phofl 
Sorry for the confusion, this should go into 2.1.2 (it#s only really relevant if you use the new option)
@meeseeksdev backport 2.1.x
DOC: Change core paths to api.typing in URLs
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

@phofl pointed out that #27522 would change the URLs in the API docs from `pandas.core` to `pandas._core`. We shouldn't be leaking the locations of the internals via the URLs. While `pandas.api.typing` is perhaps a little odd, it is at least public.

cc @jorisvandenbossche @jbrockmendel @mroeschke 

Labels:
name: Docs
name: API Design

Comments:
Good catch. Thanks @rhshadrach 
I missed this notification at the time, but remembering it now because it came up in the discussion about deprecating `core` in the call yesterday. 

Two comments:

- This will actually break all existing URL out in the wild. Generally, when we do such a renaming, we add redirects for the existing pages (we have that mechanism set up, so it's just a matter of adding all impacted pages redirects.csv, AFAIK)
- Given that changing urls is a bit annoying, we might want to give the final location a second thought. If we think about "we might want to add something specifically for docs at some point" to quote Patrick above (https://github.com/pandas-dev/pandas/issues/55632), we might better to that now, before those urls start to be used in the stable docs, to avoid breaking the urls a second time

I am all for breaking those urls once so we can remove the "core" out of those long references (which we need to deprecating `core` anyway, but those long references to "pandas.core.groupby.<..>" in the public docs have also always bothered me). 
But personally I am not sure if `pandas.api.typing` is the ideal place to put those. 

I will move the second item about what's the best location to use for the docs to the issue Patrick opened: https://github.com/pandas-dev/pandas/issues/55632
BUG: pytables with non-nano dt64
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Surfaced while implementing #55564

Labels:
name: IO HDF5
description: read_hdf, HDFStore
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @jbrockmendel
BUG: mode not sorting values for arrow backed strings
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality

Comments:
Thanks @phofl 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.1.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 bb2d2e00524fb31d08a43fc54706035f810d2489
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #55621: BUG: mode not sorting values for arrow backed strings'
```

4. Push to a named branch:

```
git push YOURFORK 2.1.x:auto-backport-of-pr-55621-on-2.1.x
```

5. Create a PR against branch 2.1.x, I would have named this PR:

> "Backport PR #55621 on branch 2.1.x (BUG: mode not sorting values for arrow backed strings)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).