BrandonFors/pandas-github-issues huggingface dataset:
DEPR offsets: rename 'Q' to 'QE'
State: closed

xref #9586, #52064

Deprecated the alias denoting quarter end frequency `"Q"` for offsets in favour of `"QE"`.
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` 


Labels:
name: Frequency
description: DateOffsets
name: Deprecate
description: Functionality to remove in pandas

Comments:
@MarcoGorelli, could you please take a look at this PR? It's a follow-up of PR: "DEPR offsets: rename 'M' to 'ME'".
seems like ci failure isn't related to my changes
looks like there's some test failures
```
 FAILED pandas/tests/io/json/test_json_table_schema.py::TestTableOrientReader::test_read_json_table_period_orient[vals0-None] - TypeError: data type 'period[QE-DEC]' not understood
```
> looks like there's some test failures
> 
> ```
>  FAILED pandas/tests/io/json/test_json_table_schema.py::TestTableOrientReader::test_read_json_table_period_orient[vals0-None] - TypeError: data type 'period[QE-DEC]' not understood
> ```

`test_read_json_table_period_orient` failed because there was a bug for conversion JSON field descriptor into pandas type for deprecated offsets frequency 'M'. We fixed it in #55581, now the test passes.

I added parameters `'Q'` and `'2Q'` into parametrization of `test_read_json_table_orient_period_depr_freq`
thanks @natmokval !
MAINT: Partially revert `np.int_` changes
State: closed

Hi!

This PR reverts some changes introduced in https://github.com/pandas-dev/pandas/pull/55369 that, after discussions, are outdated (`np.long` should be used in places that specifically refer to the C type, rather than just "default integer").


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Thanks @mtsokol
Adjust Series specific tests for string option
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data

Comments:
I'll open issues for them and link them here
Opened issues for both
BUG: Series inferring new string dtype even if dtype is given for scalar value
State: closed

This is a regression on main, so no whatsnew needed

Labels:
name: Strings
description: String extension data type and string data

Comments:
If this is a regression on main, shouldn't this target 2.2 instead?
sorry should have clarified, the regression was back ported as well
Gotcha. Thanks @phofl
TST: Fix assert_is_sorted for eas
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @phofl
BUG: Ensure "string[pyarrow]" type is preserved when calling extractall
State: closed

When calling `extractall` on an arrow string series, although the operation itself works, the returned dataframe column is of dtype `object` instead of preserving the arrow string type. This happens since the resulting dataframe was generated with the original array dtype only if it was an instance of `StringDType`. Expanded the conditional to also preserve the dtype on `ArrowDType`.

- [x] closes #53846  
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality
name: pyarrow dtype retention
description: op with pyarrow dtype -> expect pyarrow result

Comments:
Thank you @mroeschke, added
Thanks @ABizzinotto
Add env variable for infer string option
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Strings
description: String extension data type and string data
name: Stale

Comments:
Removing from milestone, since it looks like comments need to be addressed.

This also needs tests, if it were to be merged.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Resolve Heisenbug in StringHashTable._unique
State: closed

When processing an invalid Unicode string, the exception handler for UnicodeEncodeError called `get_c_string` with an ephemeral repr value that could be garbage-collected the next time an exception was raised.  Issue #45929 demonstrates the problem.

This commit fixes the problem by retaining a Python reference to the repr value that underlies the C string until after all `values` are processed.

Wisdom from StackOverflow suggests that there's very small performance difference between pre-allocating the array vs. append if indeed we do need to fill it all the way, but because we only need references on exceptions, we expect that in the usual case we will append very few elements, making it faster than pre-allocation.

- [x] closes #45929
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Strings
description: String extension data type and string data
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Given the new Cython 3.0 changes about to be merged, are there any C-API folks who might know how to take this across the finish line?
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
TYP/DOC: add HTMLFlavors type to read_html and related
State: closed

Following up on what was discussed in https://github.com/pandas-dev/pandas-stubs/pull/772, accepted html flavors are now defined in their own type instead of being a str or a generic str sequence.

Refs:
- https://github.com/pandas-dev/pandas-stubs/pull/772#discussion_r1322037456
- https://github.com/pandas-dev/pandas/issues/55059
- https://github.com/pandas-dev/pandas/pull/55076

Labels:
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @matheusfelipeog
DEPS: Bump optional dependencies
State: closed

These versions will have been around for ~1 year once 2.2 comes out

Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
I think this looks good. Is the 1 year dependency minimum new? That was 2 in the past right?
> I think this looks good. Is the 1 year dependency minimum new? That was 2 in the past right?

I bumped our optional dependencies to a 1  year min last time https://github.com/pandas-dev/pandas/pull/52552. Our required ones usually follow 2
CLN/TST: clean logic of old datetime test_indexing test
State: closed

Breaking off a change I had to make for https://github.com/pandas-dev/pandas/pull/55428, because it's actually unrelated to CoW and reduces the diff there a little bit.

I think that this test has over time, through several small changes, become quite useful / incomprehensible. So cleaning it up, and actually asserting what my understanding is that it was originally testing.

Labels:
name: Clean

Comments:
Thanks @jorisvandenbossche
CoW warning mode: warn in case of chained assignment
State: closed

Follow-up PR on https://github.com/pandas-dev/pandas/pull/55428 (only last commit is relevant here, will be reviewable once parent PR is merged), xref https://github.com/pandas-dev/pandas/issues/56019

Adding warnings for the case of chained assignment, i.e. cases for which we will issue the ChainedAssignmentError warning in the future with CoW enabled. But since this is the main breaking change of CoW, important to have a clear warning as well in the warn mode, and it is enabled in the default mode as well.

xref https://github.com/pandas-dev/pandas/issues/56019

Labels:
name: Copy / view semantics

Comments:
As for the other PR, I would warn in the default mode as well. This will need a similar system as for the other chained PRs that checks the cacher (that said, I found a bug in this mode that I am fixing now)
thx @jorisvandenbossche
DEPR: tz_convert, tz_localize, to_timestamp, to_period
State: closed

- [x] closes #52110 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Index
description: Related to the Index class or subclasses
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
PERF: read_stata for wide columns
State: closed

Before:
`59.41s call     pandas/tests/io/test_stata.py::TestStata::test_stata_119`

After: 
`1.50s call     pandas/tests/io/test_stata.py::TestStata::test_stata_119`

(This function can probably be further improved as it iterates over columns many times.)

Labels:
name: IO Stata
description: read_stata, to_stata

Comments:
Looks like tests are passing here so going to merge. Happy to follow up if needed.
Stub for Apple Numbers reader/writer
State: closed

- [x] Begins implementation of #55497 
- [x] Tests added and passed to (pandas/tests/io/test_apple_numbers.py)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Initial commit for starting a new reader and writer. Next step will be writing some tests.

Labels:

Comments:
Closing as per PR and issue comments
Update get_utc declarations
State: closed

This helps a bit with the Cython 3.0 performance; the regression here is that without noexcept Cython checks for an error each invocation of bisect_right_i8

The const qualifiers are not necessarily required, but are a nice safety add-on and only give the compiler a better chance to optimize


Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Timezones
description: Timezone data dtype

Comments:
LGTM pending green
Thanks @WillAyd
Replace is_temporal checks with Cython version
State: closed

With Cython 3.0.3 I was unable to build pandas. Looks like is_timedelta / is_datetime functions were getting mixed between static / non-static declarations. I wasn't sure why that was happening, but while investigating noted that these got ported upstream by @jbrockmendel a few years back, so I think it makes sense to leverage that instead (also clears the issue)

Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Does this affect our minimum cython/numpy versions?  (IIRC cython has a `numpy/__init__.pxd` file that got used sometimes and they wanted to get rid of it but i dont know if that ever happened or what versions are affected)
Would this allow us to unpin our Cython `<3.0.3` in our CI files?
As long as you're doing these, might as well change to use the other stuff upstreamed into numpy in numpy/numpy#16266?
> Would this allow us to unpin our Cython `<3.0.3` in our CI files?

Ah didn't realize we pinned that. I believe so; at least locally I can build the entire code base with this
> As long as you're doing these, might as well change to use the other stuff upstreamed into numpy in [numpy/numpy#16266](https://github.com/numpy/numpy/pull/16266)?

Unfortunately there is a problem where the definition we have of `NPY_DATETIMEUNIT` in pandas._libs.tslibs.np_datetime differs from what Cython offers, the main difference being the Cython version does not offer `NPY_FR_GENERIC` as a value. Our code does some comparison to that value, so we can't swap over to using `get_datetime64_unit` because Cython/C will not be happy with the difference in those enum declarations

I think that's a bug in Cython - the source file `numpy/ndarraytypes.h` does have an enum member for `NPY_FR_GENERIC` so probably just needs to be upstreamed in Cython before we can fully implement your PR
@jbrockmendel see also https://github.com/cython/cython/pull/5765/files
> With Cython 3.0.3 I was unable to build pandas. Looks like is_timedelta / is_datetime functions were getting mixed between static / non-static declarations. I wasn't sure why that was happening, but while investigating noted that these got ported upstream by @jbrockmendel a few years back, so I think it makes sense to leverage that instead (also clears the issue)

This was a bug in Cython, but I'm happy to see that this fixes it.
OK I updated the enum definition upstream in https://github.com/numpy/numpy/pull/24923#event-10675144378 so can come back around for the rest of these when that goes through. Any concern with this PR in its current state?
I'm OK with the fix. Could you also unpin Cython 3.0.3 in our files?
#55582.

I think we can unpin generally anyways since Cython 3.0.4 is out.
Thanks @WillAyd 
nice!
Revert `np.int_` and `np.uint` changes
State: closed

Hi!
In the last NumPy community meeting a decision was made that `np.int_` and `np.uint` should stay and they won't be a direct mapping to `np.long` and `np.ulong` (https://github.com/numpy/numpy/pull/24794#issuecomment-1760722889).

This PR reverts my two commits that were added.

Labels:

Comments:
Sorry for confusion! It turns out that it's fine to replace `np.long` with `np.int_` (and `np.int_` will stay), so no need to revert these changes - I will double-check the changes but this PR won't be merged.
Superseded by https://github.com/pandas-dev/pandas/pull/55540.
BUG: tslibs uncaught overflows
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @jbrockmendel
DEPR: rename ‘BM’/‘CBM’ to ‘BME’/‘CBME’ for offsets
State: closed

xref #52064

Renamed prefixes `"BM"/"CBM"` for offsets classes BusinessMonthEnd and CustomBusinessMonthEnd to `"BME"/"CBME"`.


Labels:
name: Frequency
description: DateOffsets
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thank you for reviewing this PR.
BUG: DataFrame.to_json OverflowError with np.long* dtypes
State: closed

- [x] closes #55403 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
We don't support long double as a data type and I don't think upcasting double to long double is appropriate. We should probably just raise here
@WillAyd, Thanks for identifying the issue with commit. I have updated the commit with changes suggested in above comment.
Thanks
> We don't support long double as a data type and I don't think upcasting double to long double is appropriate. We should probably just raise here

I'm curious, if pandas doesn't support these numpy types in a data frame, would it make sense to raise an exception at the time of creation or dtype assignment? There are maybe other places in the code these unsupported types would break?
@nmacholl you might want to check existing issues for any discussion, but I don't think its quite that simple. For starters assignment/creation isn't the only way those types might make there way into a DataFrame. There also exists a subset of users who may not use pandas for I/O / computations (where a lot of those limitations are imposed) versus just using it as an easy way to get labeled access to NumPy arrays. As an example, you'll see quite a few use cases in the issue log where users contain NumPy unicode dtypes in DataFrames even though pandas has almost no support for those; in such cases a DataFrame is just a labeled indexer, and I don't think we want to break that type of workflow
@gupta-paras this looks good to me aside from outstanding comments / code suggestions. @mroeschke any thoughts on your end?
Thanks @gupta-paras
REF: De-special-case _format_with_header
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
Thanks @jbrockmendel