BrandonFors/pandas-github-issues huggingface dataset:
BUG (CoW): fix reference tracking in replace_list with None
State: closed

For a specific case in `list_replace` when replacing with a non-present value with None, we were returning the block itself instead of a shallow copy (as is done in other code paths in this function), messing up the reference tracking (we are removing the block itself to avoid multiple copies when replacing multiple values).

- [x] closes #59770
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: replace
description: replace method
name: Copy / view semantics
name: backported

Comments:
@MeeseeksDev backport to 2.3.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 3e8ac12d1dacc2308b2f4c2869fa7bc2079bd323
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #59807: BUG (CoW): fix reference tracking in replace_list with None'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-59807-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #59807 on branch 2.3.x (BUG (CoW): fix reference tracking in replace_list with None)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport at https://github.com/pandas-dev/pandas/pull/59943
Update ExtensionArray.interpolate to remove outdated method of pad
State: closed

Removed the outdated method of `pad` from ExtensionArray.interpolate.

Reason:

- Found in #59749 that `pad` raises a ValueError whatever value you pass. 
- #57092) says that `pad` has been removed from DataFrame.Interpolate, which is the origin of ExtensionArray. 


Labels:
name: Docs

Comments:
Thanks @ammar-qazi
Enhance to_numeric to support hexadecimal, octal, and binary string inputs
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the PR, but it appears this is not associated with an issue. For feature requests, we require an issue to be opened and support from the core team before pursuing a PR to implement the feature. Feel free to open an issue describing the feature you'd like to have
ENH: Series.str.get_dummies() raise on string type (follow up to PR #59577)
State: closed

Follow-up on https://github.com/pandas-dev/pandas/pull/59577#issuecomment-2340442650

Labels:
name: Strings
description: String extension data type and string data

Comments:
@rhshadrach Please see latest changes
@aaronchucarroll - I believe there is an issue related to this, is that right?
I don't think there exists an issue for this. These are the requested follow-up changes to my last PR (#59577)
@rhshadrach Please see new changes
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks @aaronchucarroll and @jorisvandenbossche
Assorted backports for 2.2.x
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration
name: Python 3.13

Comments:
On phone so sorry for not linking better, but I think the DTI failures need to be partially pulled from #55901. 

If you look at the commits in the 2.3 branch I started you will see a commit where I do that
> On phone so sorry for not linking better, but I think the DTI failures need to be partially pulled from #55901.
> 
> If you look at the commits in the 2.3 branch I started you will see a commit where I do that

I think I have most of the relevant fixes picked from your PR, actually.
(and some of the rest I will fix by backporting some other PRs).

Only headscratcher for me right now is the ARM build jobs (which also happen to fail on main).
Interesting that it's only the build and not the test job, though ....

all green 🚀
Backport PR #58218: Revert "CI: Pin blosc to fix pytables"
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration

Comments:
accidentally merged this in by pushing to 2.2.x.

I'll iterate on the remaining failures.
BUG: groupby().any() returns true for groups with timedelta all NaT
State: closed

PR Summary:

This PR addresses an issue where groupby().any() returns True for groups where all Timedelta values are NaT, which is inconsistent with other dtypes (like integers and strings). For non-Timedelta dtypes, groupby().any() correctly returns False when all values in a group are null.
Reproduction:

python

import pandas as pd

# Current behavior: incorrect output for all NaT groups
df = pd.DataFrame([pd.Timedelta(1), pd.NaT]).groupby([0, 1]).any()

For other dtypes (integers and strings):

python

df = pd.DataFrame([1, None]).groupby([0, 1]).any()  # Returns False
df = pd.DataFrame(["a", None]).groupby([0, 1]).any()  # Returns False

Expected behavior for Timedelta values: groupby().any() should return False for groups where all values are NaT.
Fix Details:

    Modified the _call_cython_op function in the groupby aggregation to correctly handle Timedelta types. Now, when all values in a group are NaT, the any() method will return False.
    This ensures consistent behavior across all dtypes, including Timedelta.

Tests Added:

    A new test has been added to check that groupby().any() returns False for groups with only NaT Timedelta values and True for groups with at least one non-NaT Timedelta.

Closing:

This PR fixes the inconsistent behavior of groupby().any() for Timedelta values and improves the overall correctness of the method. Let me know if there are any additional changes needed.

Resolves: #59712

Note: I am a complete novice to open source and this is my first ever PR. I'm taking an open source systems class is college and we're suppposed to attempt to contribute to a existing project we use and I chose pandas as it is a tool that has helped me alot. So i ask for any guidance anyone can give me about any potential mistakes or improvements i can make here. Appreciate it!


Labels:
name: Bug
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Timedelta
description: Timedelta data type
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Thanks for the pull request but this is already being worked on in https://github.com/pandas-dev/pandas/pull/59750. Closing to let that contributor finish the effort, but we can reopen this if it goes stale 
Hi @mroeschke , could this be reopened?
Hi @rhshadrach, how's this?
Thanks @Petroncini!
Thanks for all the help and guidance, @rhshadrach!
BUG: avoid triggering numpy deprecation warning in assert functions for nested array with empty array/list
State: closed

I haven't added a whatsnew note, because it shouldn't change behaviour but just avoids a warning being triggered (so thought that to not warrant a mention in the release notes)


- [x] closes #59776
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Does this need a backport?
No, it's just for silencing a warning in a corner case, not worth of backporting
REF: pass dtype explicitly to _from_sequence
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @jbrockmendel
REF: pass dtype explicitly to _from_sequence inside pd.array
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @jbrockmendel
BUG/API (string dtype): return float dtype for series[str].rank()
State: closed

Noticed in https://github.com/pandas-dev/pandas/pull/59758#discussion_r1750734166

This is partially fixing a bug, because currently for cases where we actually need a float result but try to convert it back to an int, we get an error:

```python
In [58]: pd.Series([2, 1, 1]).rank()
Out[58]: 
0    3.0
1    1.5
2    1.5
dtype: float64

In [59]: pd.Series(["2", "1", "1"], dtype="string[pyarrow]").rank()
...
File ~/scipy/repos/pandas/pandas/core/arrays/string_arrow.py:445, in _convert_int_result(self, result)

File ~/scipy/repos/pandas/pandas/core/arrays/numeric.py:93, in NumericDtype.__from_arrow__(self, array)
---> 93     array = array.cast(pyarrow_type)
 ...
ArrowInvalid: Float value 1.5 was truncated converting to int64
```

But in general we should also decide what to do return here. For our default dtypes, it seems we decided in the past to simply always return float64, even for rank `method`s that could return ints. 
For the ArrowDtype, then, it was decided to keep the dtype returned by pyarrow: https://github.com/pandas-dev/pandas/pull/50264

For now, this PR updates `StringDtype` to simply always return float64, to be consistent between the pyarrow vs python storage. But we could also consider, for those newer dtypes, to actually keep the distinction between float/int results (just not always int like it is done now). 

Labels:
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality
name: Transformations
description: e.g. cumsum, diff, rank
name: backported

Comments:
> Is it worth an entry in the 2.3.0 whatsnew?

Yes, good idea, added one.
Thanks @jorisvandenbossche
Infer extensiondtype from scalar
State: closed

- [ ] closes #41848
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This adds an `is_unambiguous_scalar` (as suggested) method to `ExtensionDtype` that can be used to infer the dtype when it isn't passed into `Series` or `DataFrame` constructors.  

With these changes `Series()` can recognise `pint.Quantity`s and construct a `PintArray`. Is this a good approch? I've not done any inference in sanitize_array as its docstring doesn't mention inference. However it looks like it'd be easier to do so in there than do so in the various constructor methods.

Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Fix issue #47101
State: closed

Reapplies PR #59492

See https://github.com/pandas-dev/pandas/pull/59492#issuecomment-2338729574

--------------


Bisecting two years ago ( https://github.com/pandas-dev/pandas/issues/47101#issuecomment-1139606268 ) shows this regression was introduced in b2d54d9 in 2021. Somehow this hasn't been patched since then.

PR #48313 was supposed to address this, but the PR was closed and never merged and the bug has persisted.

Patch takes two tracks -- it first replaces the `arr == x` with [`np.equals()`](https://numpy.org/doc/1.26/reference/generated/numpy.equal.html) which will handle broadcasting in most cases by itself. For the remainder of cases, an explicit type check is done to return an array for the mask.

- [x] closes #47101
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

----------------

Has a KNOWN test failure of

> ```
> ruff-format.............................................................................................Failed
> - hook id: ruff-format
> - files were modified by this hook
> ```

but I honestly have no clue what's failing in that. Some inscrutable and poorly reported style problem?

Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
It's still a valid PR

- via Pixel 8pro (pardon insane autocorrects) -

On Wed, Oct 9, 2024, 17:07 github-actions[bot] ***@***.***>
wrote:

> This pull request is stale because it has been open for thirty days with
> no activity. Please update
> <https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request>
> and respond to this comment if you're still interested in working on this.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/59766#issuecomment-2403646772>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AABIQMJA2DVTSMYOLTW6GK3Z2XALZAVCNFSM6AAAAABN5F37KCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBTGY2DMNZXGI>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
TST (string dtype): adjust pandas/tests/reshape tests
State: closed

This addresses all xfailed `TODO(infer_string)` tests in `pandas/tests/reshape`. There are a few cases remaining, but those seem to require an actual fix and are not just a test update.


xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Strings
description: String extension data type and string data
name: backported

Comments:
Thanks @jorisvandenbossche
String dtype: remove fallback Perfomance warnings for string methods
State: closed

Closes #58581

The current PR simply removes the warning altogether, but now we have the `"mode.performance_warnings"` option, we could in theory also keep the warnings as long as they are not enabled by default. 
However, the performance warnings option right now is still on by default. So if we want to do this right now (keep the string fallback warnings but not enabled by default), we would need to change that default to something like `None` which then means True/False depending on the case .. (cc @rhshadrach)


xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Strings
description: String extension data type and string data
name: backported

Comments:
My preference would be a simple Boolean show all vs show none (ref: https://github.com/pandas-dev/pandas/issues/55385#issue-1924986475). In any case, showing warnings when the user has no viable path of better performance doesn't seem appropriate. This looks good to me.
OK, sounds good, let's then simply remove those for now.
String dtype: fix isin() values handling for python storage
State: closed

This PR adds a `StringArray.isin()` implementation (`ArrowStringArray` already has a custom `isin()`), so we can add custom processing of the passed `values` set compared to the base class implementation.

See also the discussion at https://github.com/pandas-dev/pandas/pull/58451#discussion_r1700793204 about the expected behaviour.

xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Strings
description: String extension data type and string data
name: backported

Comments:
Thanks @jorisvandenbossche
BUG (string dtype): fix inplace mutation with copy=False in ensure_string_array
State: closed

Further follow-up on https://github.com/pandas-dev/pandas/issues/54654, https://github.com/pandas-dev/pandas/pull/54687, https://github.com/pandas-dev/pandas/pull/57212, because we still had the issue when `copy=False` was passed (which happens when casting to any of the string dtypes).

The added tests failed without the actual code change.

(smaller change extracted from https://github.com/pandas-dev/pandas/pull/59685/)

xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Bug
name: Strings
description: String extension data type and string data
name: backported

Comments:
small questions, otherwise LGTM
Thanks @jorisvandenbossche
Fix/na_values_GH59303
State: closed

- [x] closes #59303 
- [x] Test added and passed: test_na_values_dict_without_dtype() in test_na_values.py 
- [x] All code checks passed.
- [x] No new type annotations.
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst`


Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
I think the two comments on the PR are now resolved, but I don't understand why some of the Unit Tests fail. I looked into the details of the fails, but they seem to be not related to this PR - If that is the case, please kindly suggest what I should do next, if there's anything needed from my end. Thank you!
Thanks @sshu2017
#59009: Added document and a test case for newlines_in_values case.
State: closed

- [x ] closes #59009 (Replace xxxx with the GitHub issue number)
- [ x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
Hi @wooseogchoi thanks for taking a look at this. Regarding the CI failures - the `Minimum Versions` failure means that whatever the lowest version allowed of pyarrow doesn't properly fix this issue. Can you check to see what version of pyarrow this might have been fixed in?

If you can identify that you can use the `pa_version_underXXpY` sentinels from pandas.compat to skip the test for versions we cannot fix (you will see uses of that already throughout the test suite)

The errors on the circleci arm build seem unrelated - they may go away if you merge in the latest changes from main
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.