BrandonFors/pandas-github-issues huggingface dataset:
ENH: add consortium standard entrypoint
State: closed

Here's an entry point to the Consortium's DataFrame API Standard

It enables dataframe-consuming libraries to just check for a `__dataframe_consortium_standard__` attribute on a dataframe they receive - then, so long as they stick to the spec defined in https://data-apis.org/dataframe-api/draft/index.html, then their code should work the same way, regardless of what the original backing dataframe library was

Use-case: currently, scikit-learn is very keen on using this, as they're not keen on having to depend on pyarrow (which will become required in pandas), and the interchange protocol only goes so far (e.g. a way to convert to ndarray is out-of-scope for that). If we can get this to work there, then other use cases may emerge

The current spec should be enough for scikit-learn, and having this entry point makes it easier to move forwards with development (without monkey-patching / special-casing)

For reference, polars has already merged this: https://github.com/pola-rs/polars/pull/10244

## Maintenance burden on pandas

**None**

I want to be very clear about this: the compat package will be developed, maintained, and tested by the consortium and community of libraries which use it. It is up to consuming libraries to set a minimum version of the dataframe-api-compat package. **No responsibility will land on pandas maintainers**. If any bugs are reported to pandas, then they can (and should) be politely closed.
All this is just an entry point to the Consortium's Standard

Tagging @pandas-dev/pandas-core for visibility. Some people raised objections (maintenance burden, naming) when asked in private, and I think I've addressed the concerns. Anything else?
Thanks 🙌 

Labels:
name: Build
description: Library building on various platforms
name: Interchange
description: Dataframe Interchange Protocol

Comments:
No objections here.
Thanks @MarcoGorelli
COMPAT: Change khcomplex64/128_t typedefs
State: closed

xref https://github.com/numpy/numpy/pull/24085

suggestion by @seberg 

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
@mroeschke 
Looks like some failures
```
FAILED pandas/tests/io/formats/test_printing.py::test_ser_df_with_complex_nans[True-data6-output6] - AssertionError
FAILED pandas/tests/io/formats/test_printing.py::test_ser_df_with_complex_nans[False-data6-output6] - AssertionError
```
Can't reproduce this locally, but it seems that `float real` of `complex(1.2, nan)` is returning NaN instead of 1.2. Any ideas @WillAyd 
I think it might be a Python 3.12 issue. The regular numpy dev build doesn't fail.
> I think it might be a Python 3.12 issue. The regular numpy dev build doesn't fail.

I think numpy dev has cached dependencies so maybe the changes haven't caught up
Looks like this change subtly changed how `maybe_convert_objects` infers complex scalars, but given that the failing test is a net new fix in 2.1 and the failure is an edge case with nan, I think the status quo is better that the behavior in 2.0 so merging as is.
BUG: pd.array([]) should return masked array
State: closed

- [x] closes #54371
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @topper-123
REF: remove test_accumulate_series_raises
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
```
pandas/tests/extension/test_arrow.py:358: error: Item "dtype[Any]" of "dtype[Any] | ExtensionDtype" has no attribute "pyarrow_dtype"  [union-attr]
pandas/tests/extension/test_arrow.py:358: error: Item "ExtensionDtype" of "dtype[Any] | ExtensionDtype" has no attribute "pyarrow_dtype"  [union-attr]
```
Thanks @jbrockmendel
CLN: Remove None check in attrs property lookup
State: closed

`self._attrs` is always initialized to an empty dict per https://github.com/pandas-dev/pandas/blob/c93e8034a13d3dbe2358b1b2f868a0d54d1034a7/pandas/core/generic.py#L275

The attribute `_attrs` is only witten to in two other places

a) in the attrs.setter property (which enforces dict as well)
b) in `__setstate__`, which takes whatever the state is:
   https://github.com/pandas-dev/pandas/blob/c93e8034a13d3dbe2358b1b2f868a0d54d1034a7/pandas/core/generic.py#L2129
   AFAICS (including code history) I do not see a reason that this could be None.
   But if we wanted to be very defensive, we should do the dict enforcing here.



Labels:
name: metadata
description: _metadata, .attrs

Comments:
>  But if we wanted to be very defensive, we should do the dict enforcing here.

This would be a good change for this PR
I've now included the dict-enforcement in `__setstate__`.
Thanks @timhoffm
BUG: is_list_like returns True even if __iter__ raises an error
State: closed

Changed so c_is_list_like() returns false if __iter__() throws exception

- [x] closes #54176
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
@mroeschke Is removing the implementation of is_list_like as it is and going in the direction of a handful of explicitly supported types the correct direction to go?
`is_list_like` is public so it can't be outright removed, but slowly internal usages should move away from `is_list_like`.

IOW, I would recommend working on another issue in the meantime until this issue is more settled
Thanks for the PR but going to close until there's more direction on the original issue
Added pivot table test for explicit datetime types Issue#43574
State: closed

- [ ] closes #43574
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Thanks @kvn4
DOC: Fixing EX01 - Added examples
State: closed

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Towards https://github.com/pandas-dev/pandas/issues/37875

Labels:
name: Docs

Comments:
Just a merge conflict otherwise looks good
Thanks @DeaMariaLeon
TST: Add tests to ensure proper conversion of datetime64/timedelta64/timestamp/duration to and from pyarrow
State: closed

- [ ] closes #54191 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).



Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
As per [this issue from the pyarrow repository](https://github.com/apache/arrow/issues/33321) there was error in convert_dtypes and pa.table()/pd.to_pandas() in which pyarrow would always return [ns] as unit regardless of associated metadeta. I have added tests to ensure the conversion always preserves associated unit. 

Pyarrow has since fixed and closed the issue in their current main branch (dev version 13.0), and so when it is publicly available the if conditionals mentioned in #54191 can be considered for removal with these tests to ensure functionality remains the same after their removal. 
I've fixed the test to match the issue and put it in the proper location, but I'm still having a failing CI "minimum versions" and I can't pinpoint why if anyone could help with that.
Might help to merge main again
I apologize for all the commits, git is new to me
Thanks @Julian048
DOC: Fixing EX01 - Added examples
State: closed

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Towards https://github.com/pandas-dev/pandas/issues/37875

Sphinx acting weird locally :-/ 

Labels:
name: Docs

Comments:
~~Doc build failed~~
It's green
Thanks @DeaMariaLeon
DOC: Fixing EX01 - Added examples
State: closed

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Towards https://github.com/pandas-dev/pandas/issues/37875

Labels:
name: Docs

Comments:
/preview
No preview found for PR #54350. Did the docs build complete?
/preview
No preview found for PR #54350. Did the docs build complete?
/preview
No preview found for PR #54350. Did the docs build complete?
> No preview found for PR #54350. Did the docs build complete?

how odd, it shows here https://github.com/pandas-dev/pandas/actions/runs/5725144896?pr=54350
hey, this looks correct! 🙌 

![image](https://github.com/pandas-dev/pandas/assets/33491632/f8b90584-634a-4d40-a8a4-18aefc773b18)

It's green
Thanks @DeaMariaLeon
DEPR offsets: rename 'Q' to 'QE'
State: closed

xref #52064

Deprecated the alias 'Q' for offsets in favour of 'QE'.


Labels:
name: Frequency
description: DateOffsets
name: Deprecate
description: Functionality to remove in pandas
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
restructured drop method return value defination
State: closed

- [x] closes #54319 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @aliasgar55
REF: dont patch assert_frame_equal
State: closed

This is the only place where we patch assert_frame_equal in the tests.  I think this pattern is pretty useless since it indicates that the relevant EA test wouldn't work for a user who didn't _also_ patch assert_frame_equal.  So we should merge this and then follow up to get rid of the patching-assert_frame_equal pattern altogether.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @jbrockmendel
ENH: allow opt-in to inferring pyarrow strings
State: closed

cc @phofl 

Enabling this at the top of conftest.py breaks 4290 tests.

Labels:
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality

Comments:
I'm not as invested as you in getting this in for 2.1.  Want to take over this PR?
I have some trouble pushing to other forks, opened #54430 with tests and removed the comments
Closing in favor of #54430
REF: de-duplicate extension tests
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @jbrockmendel
DOC: Move performance improvements in whatsnew
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

The large reshuffle was from the read_csv performance note breaking our parsing

Labels:
name: Docs
name: Clean

Comments:
Thanks @rhshadrach
TST: fix Decimal constructor xfail
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @jbrockmendel
REF: remove overriding assert_extension_array_equal
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @jbrockmendel
DEPS: Bump cython 3.0
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Dependencies
description: Required and optional dependencies

Comments:
Looks pretty good. Are all of these changes required to make Cython 3.0 work or are some of the signature changes general cleanups that made sense to bundle?

@Dr-Irv FYI in case this impacts pandas-stubs
> Looks pretty good. Are all of these changes required to make Cython 3.0 work or are some of the signature changes general cleanups that made sense to bundle?
> 
> @Dr-Irv FYI in case this impacts pandas-stubs

Yeah, it turns out our stubs are horrendously outdated. I guess Cython 3.0 exposes the annotation information, so stubtest is complaining now.
> 
> @Dr-Irv FYI in case this impacts pandas-stubs

We're not that good in keeping the PYI files in the pandas source and the PYI files in `pandas-stubs` in sync.  There's some related discussion here:  https://github.com/pandas-dev/pandas-stubs/issues/128

One of the issues here is that the pandas source has things in its PYI files for "private" methods - i.e., methods and classes that are not documented but are used to type check the pandas source.  The philosophy on pandas-stubs is to only provide stubs for documented methods and classes - although we may be missing some documented methods/classes that should be in the pandas-stubs, and there are also ones that are in pandas-stubs that are not documented.  Cleaning that up is a long term goal.
Do we really want to include this bump in the 2.1.0 release? AFAIK Apache Arrow will adapt Cython 3.0.0 in their 14.0.0, but they do not release 13.0.0 yet, meaning the integration with Apache Arrow will not work, right? 
> Do we really want to include this bump in the 2.1.0 release? AFAIK Apache Arrow will adapt Cython 3.0.0 in their 14.0.0, but they do not release 13.0.0 yet, meaning the integration with Apache Arrow will not work, right?

We don't compile against Apache Arrow (only numpy). All integration with arrow is done through Python IIRC.
Nice thanks @lithomas1 
very cool to see this in!
REF/TST: handle boolean dtypes in base extension tests
State: closed

Fixes a handful of xfails in the pyarrow tests, avoids special-casing in a bunch of the BooleanArray tests.

Labels:
name: Refactor
description: Internal refactoring of code
name: Testing
description: pandas testing functions or related to the test suite
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @jbrockmendel
details on weekday argument for DateOffset docs
State: closed

- [ ] towards #52650
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.


Labels:
name: Docs
name: Frequency
description: DateOffsets

Comments:
Hey @Detoro . Can you try commenting ``` pre-commit.ci autofix ```?
pre-commit.ci autofix
/preview
No preview found for PR #54332. Did the docs build complete?
> thanks for your PR
> 
> this addresses part of the issue - the other part is about the link
> 
> in `See also`, could you replace `dateutil.relativedelta.relativedelta` with https://dateutil.readthedocs.io/en/stable/relativedelta.html please?
> 
> thanks

We were planning to do this in a follow up PR. Do you prefer if both parts are done in separate PRs, or a single PR?
/preview
No preview found for PR #54332. Did the docs build complete?
pre-commit.ci autofix
@mroeschke could you take a look?
why did you close this?
my bad @MarcoGorelli
REF: share IntegerArray/FloatingArray extension tests
State: closed

Before long I'd like to merge the BooleanArray tests into these too, basically mirroring what we have for ArrowEA.

Bigger picture I think these tests could use some attention to make it so EA authors basically only have to write the fixtures and everything else Just Works.

Labels:
name: Refactor
description: Internal refactoring of code
name: Testing
description: pandas testing functions or related to the test suite
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Nice +1 toward this direction
COMPAT: Pyarrow supports div with duration
State: closed

- [ ] closes #54315 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Arrow
description: pyarrow functionality

Comments:
LGTM pending greenish
Yeah agreed. I'll reopen the issue and relabel it a testing issue
Thx
DOC : Updated the comparison with spreadsheets with Missing Values
State: closed

- [x] part of https://github.com/pandas-dev/pandas/issues/38990
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.


Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: respect allow_copy=False in interchange.from_dataframe
State: closed

On `main`, the following doesn't raise:
```python
import pyarrow as pa
import pyarrow.compute as pc

n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]])
names = ["n_legs"]
table = pa.table([n_legs], names=names)

pd.api.interchange.from_dataframe(table, allow_copy=False)
```

Labels:
name: Interchange
description: Dataframe Interchange Protocol

Comments:
Nice thanks @MarcoGorelli