BrandonFors/pandas-github-issues huggingface dataset:
API/TST: expand tests for string any/all reduction + fix pyarrow-based implementation
State: closed

While working on adding any/all support for the object-dtype version of the StringDtype in https://github.com/pandas-dev/pandas/pull/58451, I bumped into some issues with the current testing and implementation.

It seemed that our current testing is a bit limited, so I expanded the existing test to cover more cases and to be parametrized over all string dtype variants (including plain object dtype).

But that uncovered some issues with the current implementation of the pyarrow-backed version:

- `any` could return `pd.NA` in case of `skipna=False`, while for this version of the string dtype (not using Kleene logic), the result should always be True or False. This is an easy fix (we currently were only filling missing values in case of `all`, also do this for `any`)
- **How to treat missing values?** As truthy or falsey? The current pyarrow-based implementation explicitly filled the missing values with False, but the [documentation](https://pandas.pydata.org/docs/reference/api/pandas.Series.any.html) says _"If skipna is False, then NA are treated as True, because these are not equal to zero."_ 
  I find that a bit strange (my first expectation would be to treat missing values as False), but it _is_ explicitly documented, and follows the behaviour of current object dtype (eg  `pd.Series(["a". np.nan], dtype=object).all(skipna=False)` gives True, not False.

So for now this PR updated the pyarrow implementation to follow the documented behaviour.

- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: API Design
name: Strings
description: String extension data type and string data
name: Reduction Operations
description: sum, mean, min, max, etc.
name: backported

Comments:
In 3.0, adjacently `.idxmax/min` will raise a `ValueError` if `skipna=False` and a NA value is encountered. There _could_ be an API behavior argument to align with that behavior in the future.

Besides that point, I would also agree with your initial expectation that NA values are falsey.
> In 3.0, adjacently `.idxmax/min` will raise a `ValueError` if `skipna=False` and a NA value is encountered. There _could_ be an API behavior argument to align with that behavior in the future.

For `any`/`all`, I think in the future we don't have to raise about missing values, but we can use Kleene logic and return pd.NA in some cases (although that might then just give an error downstream in your code ..).

That's the behaviour that is currently implemented for the nullable BooleanDtype, and if we plan to stick to that behaviour, I would say it's not worth to change the current NaN-based dtypes to start raise an error for NaNs (and we can preserve the strange "NaN is considered as True because it is not equal to 0" for now).


Thanks @jorisvandenbossche
Use bitmask for MaskedArray mask
State: closed

Not 100% working yet but wanted to open for discussion to see if it is worth trying to finish. This replaces our current bytemask with a custom bitmask, but does so in a separately library. The advantage of the separate library is that the CI setup can be customized a bit further, with ASAN/gtest tooling that would be hard or awkward to set up in pandas. The disadvantage is that the separate library has a sole purpose of fitting the needs of pandas, as a lot of the logic built for it tries to merge NumPy indexing with memory backed by nanoarrow; I don't see that being a wide use case

@jbrockmendel @jorisvandenbossche @phofl in case of interest

Labels:
name: Stale

Comments:
Git repo of dependency: https://github.com/WillAyd/pandas-bitmask
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Not actively working on this - let's clear the queue
With something like this could we pass the bitmask directly to pa.array for the `mask` keyword and have it be zero-copy?
Yes this would be zero copy.

Unfortunately I haven't looked at this in quite some time, and don't have any firm plans to restart. But happy to offer guidance if someone else wants to try
BUG: Handle floating point boundaries in `qcut`
State: closed

- [x] closes #59355 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.

Panda's `quantile` uses NumPy's `percentile`, so it has to convert the quantile (between 0 and 1) to a percentile (between 0 and 100). However, `np.percentile` itself uses NumPy's quantile code, so `np.percentile` [converts from a percentile back into a quantile](https://github.com/numpy/numpy/blob/1d49c7f7ff527c696fc26ab2278ad51632a66660/numpy/lib/_function_base_impl.py#L4271). Together, these two operations can introduce a slight floating point error when the quantile inputs are multiplied then divided by 100, which is not a power of 2. This PR changes Panda's quantile code to directly use NumPy's quantile code, which was [not available](https://github.com/numpy/numpy/pull/10199) when the original [Pandas quantile code was written](https://github.com/pandas-dev/pandas/pull/14536).

This PR also changes how `qcut` picks quantiles when asked to split into a fixed number of quantiles. Some quantiles, such as 5/7, can't be represented as floats and have to be rounded to the nearest representable number. If that involves rounding down, then the level that defines the upper bound of the 5/7 quantile is incorrectly assigned to the 6/7 quantile. This PR changes the quantiles to round up rather than to the nearest when picking a floating point representation.




Labels:
name: cut
description: cut, qcut

Comments:
Thanks @rob-sil
DOC: Enforce Numpy Docstring Validation | pandas.api.extensions.ExtensionArray
State: closed

- xref https://github.com/pandas-dev/pandas/issues/58539

fixes:
```
pandas.api.extensions.ExtensionArray._pad_or_backfill 
pandas.api.extensions.ExtensionArray._reduce
pandas.api.extensions.ExtensionArray._values_for_factorize
pandas.api.extensions.ExtensionArray.astype
pandas.api.extensions.ExtensionArray.dropna
pandas.api.extensions.ExtensionArray.dtype
```



Labels:
name: Docs

Comments:
Thanks @ivonastojanovic
CI: Update to cibuildwheel 2.20.0
State: closed

[cibuildwheel](https://github.com/pypa/cibuildwheel) [2.20.0](https://github.com/pypa/cibuildwheel/releases/tag/v2.20.0) uses the ABI stable Python 3.13.0rc1 and build Python 3.13 wheels by default, which allows removing the `CIBW_PRERELEASE_PYTHONS` flag.

Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration

Comments:
About 80 of these types of errors happen, on Python 3.13 only:
      
```
  >       assert isinstance(left, ExtensionArray), "left is not an ExtensionArray"
  E       AssertionError: left is not an ExtensionArray
```
Any idea what causes these?
It seems similar errors are [going on](https://github.com/pandas-dev/pandas/actions/workflows/wheels.yml?query=branch%3Amain) on the main branch ([example](https://github.com/pandas-dev/pandas/actions/runs/10242118536/job/28331368285)).

Edit: It seems this was the [last passing wheel build](https://github.com/pandas-dev/pandas/actions/runs/10105253109), which ran on July 26th. It did include successful Python 3.13 wheel builds.

So the errors are probably unrelated to this PR.
Thanks @EwoutH 
Backported in https://github.com/pandas-dev/pandas/pull/59835
ENH: pd.concat with keys and ignore_index=True should raise 
State: closed

- [x] closes #59274 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Thanks @jahn96
DOC: Fix pandas.Series.str.index RT03
State: closed

- [x] xref https://github.com/pandas-dev/pandas/issues/58504

Fixes:
`pandas.Series.str.index RT03`
`pandas.Series.str.rindex RT03`

Labels:
name: Docs
name: Series
description: Series data structure
name: Accessors
description: accessor registration mechanism (not .str, .dt, .cat)

Comments:
Thanks @KatsiarynaDzibrova
DOC: Fix pandas.Series.str.get RT03,SA01
State: closed

- [x] xref https://github.com/pandas-dev/pandas/issues/58504

Fixes:
`pandas.Series.str.get RT03,SA01`


Labels:
name: Docs
name: Series
description: Series data structure
name: Accessors
description: accessor registration mechanism (not .str, .dt, .cat)

Comments:
Thanks @KatsiarynaDzibrova
DOC: Fix Pandas.series.str.fullmatch RT03
State: closed

- [x] xref https://github.com/pandas-dev/pandas/issues/58504

fixes `pandas.Series.str.fullmatch RT03`


Labels:
name: Docs
name: Series
description: Series data structure
name: Accessors
description: accessor registration mechanism (not .str, .dt, .cat)

Comments:
Thanks @KatsiarynaDzibrova
DOC: Fix RT03 for pandas.Series.str.find
State: closed

- [x] xref https://github.com/pandas-dev/pandas/issues/58504

Fixes:
`pandas.Series.str.find RT03`
`pandas.Series.str.find RT03`

Changes description to make difference between `find` and `rfind` more clear

Labels:
name: Docs
name: Series
description: Series data structure
name: Accessors
description: accessor registration mechanism (not .str, .dt, .cat)

Comments:
Thanks @KatsiarynaDzibrova
TST: Test non-nanosecond datetimes in PyArrow Parquet dataframes
State: closed

This PR tests that the current version of pandas supports non-nanosecond PyArrow dataframes without using [timestamp_as_object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.to_pandas) param.

- [x] closes https://github.com/pandas-dev/pandas/issues/49236
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@natmokval @WillAyd Please review when you have time. Thank you!
All tests pass in my Docker container, but the CI build fails. I'll look into it.
Update: I see other people's PRs fail with similar errors. Is there a problem with CI?
Also this does potentially solve the example from the OP in #49236 , although we may not want to make this as closing that. Not sure if there is a need for the requested keyword beyond this one example
@WillAyd Thank you very much for your review!

> I'm wondering if this is that much different than the `test_read_dtype_backend_pyarrow_config` that already exists?

While `test_read_dtype_backend_pyarrow_config` tests microseconds as well, it uses a datetime that is in the normal nanosecond date range (1678 - 2262 as mentioned in PyArrow docs). My test uses the date 1600-01-01 from the issue, which is outside the normal range. 
Let me know if you think it's redundant and `test_read_dtype_backend_pyarrow_config` covers everything.
Sounds good and thanks for explaining. I think it's ok to add - generally more tests are good. 
Whoops sorry for the bad guidance on ensure_clean()
@mroeschke thanks for the suggestions!
In the failed environment, pandas 1.5.3 is used, which fails with the same error from the original issue ticket. 
Is using an old version of pandas in that environment intended for testing backwards compatibility? Should I somehow isolate my test to run in pandas 2 environments only? @WillAyd 
@mroeschke could you please help with my question above? Thanks! 
You'll probably need specify the `minversion` argument in `pytest.importorskip("pyarrow")` call. There's probably a version of pyarrow where this behavior was fixed
Thanks @EduardAkhmetshin
ENH: Allow Iterable[Hashable] in drop_duplicates
State: closed

- [x] closes #59237 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: duplicated
description: duplicated, drop_duplicates

Comments:
Also - can you give this PR an informative title. Something like `ENH: Allow sets in drop_duplicates`.
Thanks @kirill-bash!
Appreciate your help, @rhshadrach!
ENH: Implement option 'truncate' of argument `if_exists` in `DataFrame.to_sql`
State: closed

- [x] closes #37210 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: IO SQL
description: to_sql, read_sql, read_sql_query
name: Stale

Comments:
Hi @erfannariman o/.
You were the only member of @pandas-dev to participate in the discussion of the issue - and that's why I @ you here :).
The PR is open for a while now and I just wanted to know if:
1. there's still interest in this feature;
2. Is there something I can do to help the reviewing process ?

Thanks in advance.
> Hi @erfannariman o/. You were the only member of @pandas-dev to participate in the discussion of the issue - and that's why I @ you here :). The PR is open for a while now and I just wanted to know if:
> 
> 1. there's still interest in this feature;
> 2. Is there something I can do to help the reviewing process ?
> 
> Thanks in advance.

I reacted in the PR. Let's wait a couple days to see if we get some responses from others. If not, feel free to ping me again.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
The best option here is to close this PR. The community and the maintainers couldn't reach consensus over the feature in a timely manner - despite my efforts. I will be happy to work in this feature once we achieve that.
String dtype: use 'str' string alias and representation for NaN-variant of the dtype
State: closed

As we discussed for PDEP-14, the idea is that we would keep "string" for the NA-variant of the string dtype (for backwards compatibility), and then to have a distinguishing string representation and alias, use "str" for the default NaN-variant.

This PR updates StringDtype to make that distinction.

One point of discussion that came up while implementing this, what to do with `dtype == "string"` ? 

We have quite some places internally in the code where we use this to check for a StringDtype in general (instead of something like `isinstance(dtype, StringDtype)`). But should that exclusively return True for the NA-variants of the StringDtype? (to match the string alias/repr) 
For now I kept a special case to let this return True, but so I could also update all cases in our own code where we currently use this pattern, and change `dtype == "string"` to return False for the NaN-variant (`dtype == "str"` returns True for the NaN variant, and False for the NA variant)


xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Strings
description: String extension data type and string data
name: backported

Comments:
> One point of discussion that came up while implementing this, what to do with `dtype == "string"` ?

While chatting with Patrick/Brock/Will, we were thinking to for now keep this returning True also for the NaN-variants (as the current PR was doing)
make dtype to be ignored in constructors if data is Series or DataFrame
State: closed

The note in the docs added in #59300 wasn't sufficient: the dtype kwarg was ignored in certain cases. Set it to None if creating Series from another Series and the same for DataFrames. See GH pandas-dev#59060.

Request review from @mroeschke and @jorisvandenbossche

- [x] closes #59060
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
No. Dtype keyword should not be silently ignored
@jbrockmendel in the original PR I added a warning initially, though I was told that it's too noisy and a line in the docs [should be enough](https://github.com/pandas-dev/pandas/pull/59300#pullrequestreview-2194500140). Thus this PR just makes the code to act as said in the docs since the original PR is merged already.
That PR was wrong. The keyword should never be silently ignored.
[As I said in the issue](https://github.com/pandas-dev/pandas/issues/59060#issuecomment-2252256406) my PR should be reverted, I don't like the idea of silently ignoring the keywords. Though since the last one was approved I just decided to try to make it consistent. BTW ideas and suggestions are welcomed in the #59060.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DOC: Add 'See Also' and 'Parameters' for pandas.api.indexers.VariableOffsetWindowIndexer
State: closed

Add 'See Also' and 'Parameters' for pandas.api.indexers.VariableOffsetWindowIndexer

- xref https://github.com/pandas-dev/pandas/issues/58577 fixes pandas.api.indexers.VariableOffsetWindowIndexer


Labels:
name: Docs

Comments:
Thanks @ivonastojanovic
String dtype: restrict options.mode.string_storage to python|pyarrow (remove pyarrow_numpy)
State: closed

We allowed to set `"pyarrow_numpy"` as a value to the `mode.string_storage` option.

I assume this was mostly done for testing purposes, because we explicitly documented that this option would be ignored when enabling the `future.infer_string` option (i.e. when "pyarrow_numpy" is being used by default).

In context of the new string dtype (PDEP-14), I think we should remove that option again (since we will generally deprecate/remove "pyarrow_numpy"). Given the above (not documented), I opted for directly removing it from the option instead of first deprecating it (for the StringDtype constructor / string alias, we will first deprecate it). But I did add a clarification to the option error message in case someone actually did use it.

xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Strings
description: String extension data type and string data
name: backported

Comments:
Thanks @jorisvandenbossche
TST (string dtype): replace string_storage fixture with explicit storage/na_value keyword arguments for dtype creation
State: closed

The `string_storage` fixture is currently still parametrized as "python", "pyarrow" and "pyarrow_numpy". In light of the PDEP-14 work, we need to get rid of the "pyarrow_numpy" option (since this is replaced by `storage="pyarrrow", na_value=np.nan`). 

This PR introduces a new fixture to replace the current one for a subset of its use cases, where we currently use it to create the different variants of the string dtype. 

(we might also want to have a direct top-level fixture for those string dtype, but: 1) for those tests, the fixture should be named `dtype` and so easier to keep that defined within the file, 2) we already have a top-level `any_string_dtype` fixture, but that also includes `object` dtype)


xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data
name: backported

Comments:
Thanks @jorisvandenbossche
fix: show correct error message when invalid period alias is passed to to_timestamp
State: closed

addresses   https://github.com/pandas-dev/pandas/issues/58974#issuecomment-2164265446

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Frequency
description: DateOffsets

Comments:
thanks @MarcoGorelli for fixing this. Looks good to me.
Thanks @MarcoGorelli
add sphinx-copybutton configurations to exclude prompts
State: closed

- [ ] closes #59369



Labels:

Comments:
It is easy to combine the extensions `sphinx_copybutton` and `sphinx_toggleprompt`, but `sphinx_toggleprompt` doesn't support the `ipython` layout. See a minimal example I made with the code from the [beginners section](https://pandas.pydata.org/docs/user_guide/index.html#how-to-read-these-guides).

![grafik](https://github.com/user-attachments/assets/82f75f74-7e3e-41a7-8d84-774f80b77eb1)

If the regular expression from `sphinx_copybutton` isn't activated, it is not possible to copy the code from the first block to the clipboard without the output lines. This is not user friendly.

Combining these two extensions for a supported layout will result in two clicks for the user to copy the code. And in my humble opinion this is one click to much.

@mroeschke Could you please give me some feedback to the current version. Otherwise I am about to close this PR, because it is out of my personal scope.