BrandonFors/pandas-github-issues huggingface dataset:
Added an Array Extension which include missing or NAN values while wo…
State: closed

…rking with interger data types.

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the PR, but it lacks detail about the intended change. If the PR title is an indication of the proposed enhancement, it's best discussed in an issue first so closing until that issue has been created
TST: annotations, fix test_invert
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @jbrockmendel
BUG: _validate_setitem_value fails to raise for PandasArray #51044
State: closed

- [x] closes #51044 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Stale

Comments:
I would like to know if I'm going in the right direction/if the logic needs to be changed
> I would like to know if I'm going in the right direction/if the logic needs to be changed

A bunch of tests are failing on the CI.  This is the first place to look for potential problems.


Hi @mroeschke are you able to provide any guidance here?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
ENH: add cummax/cummin/cumprod support for arrow dtypes
State: closed

- [x] closes #52085
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.

un-xfails 186 tests which saves a chunk of time in `test_arrow.py`

The new pyarrow cumulative functions are only available starting in pyarrow version 13 which is not yet released but is currently tested in CI via the pyarrow nightly job. We could hold off on merging until pyarrow 13 is released if there are any concerns.




Labels:
name: Enhancement
name: Arrow
description: pyarrow functionality

Comments:
> un-xfails 186 tests which saves a chunk of time in test_arrow.py

nice!

i'd also be OK with implementing it for pyarrow>=13 and leaving it xfailed for earlier versions
I would also be okay backporting this to 2.1
Thanks @lukemanley
TST: use single-class pattern for Arrow, Masked tests
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays
name: Arrow
description: pyarrow functionality

Comments:
Thanks @jbrockmendel
TST: use single-class pattern for Categorical tests
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Categorical
description: Categorical Data Type

Comments:
Thanks @jbrockmendel
BUG: inverting object dtype containing bools
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Motivation: when trying to deprecate the inplace keyword in `where` in favor of using `mask` with a flipped condition, i found some tests failed because we pass a object-dtype frame of bools and inverting that frame gives weird results.  This is because the python behavior is weird:

```
>>> ~True
-2
>>> ~False
-1
```

This has the impact that if we have a bool-dtype Series and invert it, the behavior doesn't pointwise-match the object-dtype behavior.  This PR changes the object-dtype behavior to what I think is more reasonable. 

If someone were to complain about this diverging from the python/numpy behavior, I'd understand.


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Stale

Comments:
Was there an existing issue about this? I found https://github.com/pandas-dev/pandas/issues/51567 but not sure if this case was also specifically discussed somewhere
> Was there an existing issue about this?

I dont think we have one, but there is https://github.com/numpy/numpy/issues/23926, also i think i mentioned it on slack at some point
Can we raise/do nothing instead to follow future Python versions? Python 3.12 deprecated this.

https://docs.python.org/3.12/whatsnew/3.12.html#:~:text=The%20bitwise%20inversion,gh%2D103487.)
I wasnt aware of that python deprecation.  That makes doing nothing more attractive.  But then that'll mean for bool dtype the vector behavior wont match the pointwise (raising) behavior.  Deprecating `~` for bool dtype would seem really weird though.  do we just live with the difference?

(note to self: open an issue about deprecating Timestamp utcfromtimestamp, utcnow as they are deprecated in the stdlib) 
Hm numpy should also be hitting the same issue. I'm curious what they think about the issue.

cc @seberg
We don't have object dtype as often, especially for bools.  So I think we just say that this is awkward if you have Python bools and do whatever Python does.  It is nice that Python deprecates this.

NumPy doesn't do indexing with object dtype, we would convert to bool or integer first.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
REF: remove unnecessary Block specialization
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Broken off from a PR implementing optional 2D support for EAs.  I can't detect any performance difference locally, want to get this in and check the asvs in a few days.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
```
mypy (stubtest)..........................................................Failed
- hook id: mypy
- duration: 73.79s
- exit code: 1

/home/runner/work/pandas/pandas/scripts/run_stubtest.py:24: UserWarning: stubtest may fail as 2.1.0rc0+22.gd7b43a57b6 is not a dev version. Please install a pandas dev version or see https://pandas.pydata.org/pandas-docs/stable/development/contributing_codebase.html#validating-type-hints on how to skip the stubtest
  warnings.warn(
pandas._libs.internals.NDArrayBackedBlock is not present at runtime
pandas._libs.internals.NumpyBlock is not present at runtime
pandas._libs.internals.SharedBlock is not present at runtime
```
mypy complaints fixed + green
Thanks @jbrockmendel
fix #54564
State: closed

- [x] closes #54564
- [x] Tests added and passed
- [x] All code checks passed
- [x] Added type annotations to new arguments/methods/functions: *No new functions.*
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file: *-> may this fix make it into a potential v2.0.4 instead?*


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: IO Excel
description: read_excel, to_excel

Comments:
Note that I had to add a new `test6.xls` file to support the test, as pandas doesn't write `.xls` files, and I didn't want to add `xlwt` to the dev requirements just to generate the file on the fly. However, all `test1` to `test5` test-files exist in various excel file variants. Should `test6` be available in other versions too? How have those previous test files been generated?
Looks like the added test needs to handle Window's 32 bit default
Yea, just noticed that. @burnpanck - NumPy has a bit of an oddity where ints default to 32 bit on Windows (they're fixing this in 2.0 I believe). Can you specify `dtype="int64"` when you create `expected` in your test.

https://github.com/pandas-dev/pandas/actions/runs/5902921221/job/16011817499?pr=54567#step:5:1041

Edit: I forgot this has a mix of columns. You can use astype on the integer one.
Thanks @burnpanck
ENH: support Index.any/all with float, timedelta64 dtypes
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Match the Series behavior.

Labels:
name: Index
description: Related to the Index class or subclasses
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Thanks @jbrockmendel
Named agg tuples Issue #18220
State: closed

- [ ] closes #18220
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Okay noted

On Mon, Sep 11, 2023 at 11:41 AM Matthew Roeschke ***@***.***>
wrote:

> Thanks for the pull request, but it appears to have gone stale. If
> interested in continuing, please merge in the main branch, address any
> review comments and/or failing tests, and we can reopen.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/54563#issuecomment-1714236239>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AUGCC76LHSKEVYOJ52VPBMDXZ45MJANCNFSM6AAAAAA3RRSQTU>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
REF: remove special-casing for IntervalDtype in Index._raise_if_missing
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: Interval
description: Interval data type

Comments:
Thanks @jbrockmendel
Fixed unexpected np.nan value with reindex on pd.series with pd.Inter…
State: closed

- [x] closes #51826
- [x] [Tests added and passed]
- [x] All [code checks passed]

The test added in this issue is failing on Linux-32-bit. This is caused by another issue as discussed in #23440. The description of the the error is also described below in the comments. The test is marked xfail for 32-bit system in this PR.   

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Interval
description: Interval data type

Comments:
I have tried to enforce the 64-bit integers on the index arrays.
```
def test_interval_index_reindex_behavior(base, expected_result):
    # GH 51826
    left = np.arange(base, dtype=np.int64)
    right = np.arange(1, base + 1, dtype=np.int64)
    d = Series(
        range(base),
        index=IntervalIndex.from_arrays(left, right)
    )
    result = d.reindex(index=[np.nan, 1.0])
    tm.assert_series_equal(result, expected_result)
 ```
 However, the 32 bit tests still seems to fail. The specific error is 
 ```
 FAILED pandas/tests/indexing/interval/test_interval.py::TestIntervalIndexInsideMultiIndex::test_interval_index_reindex_behavior[101-expected_result2] - TypeError: Cannot cast array data from dtype('int64') to dtype('int32') according to the rule 'safe'
FAILED pandas/tests/indexing/interval/test_interval.py::TestIntervalIndexInsideMultiIndex::test_interval_index_reindex_behavior[1010-expected_result3] - TypeError: Cannot cast array data from dtype('int64') to dtype('int32') according to the rule 'safe'
```

And the error also points to `line:196` on `pandas/_libs/intervaltree.pxi` which is this return statement.
```
cdef take(ndarray source, ndarray indices):
    """Take the given positions from a 1D ndarray
    """
    return PyArray_Take(source, indices, 0)
```
 Do anyone have ideas on solving this unexpected casting issue?
 
Update: This error is already discussed in #23440. 
Thanks @raj-thapa
DOC: whatsnew 2.1.0 refinements
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Now there's a fun PR number!

Labels:
name: Docs

Comments:
/preview
No preview found for PR #54545. Did the docs build complete?
Thanks @rhshadrach
refactor grouper __iter__
State: closed

- [x] closes #52468 
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Removed the `__iter__` overload that was introduced in #51414. The logic accessing the `grouping_vector` for was introduced #53088

Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby

Comments:
Thanks @taytzehao!
Create EA from groupby
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: 54445 interferes with EAs that support complex128
State: closed

When a loc indexer goes to create a fresh array to hold values from an Extension Array, it makes the array allocation based on the na_value of the EA, but that na_value may be smaller than the size of the things that the EA can hold (such as complex128).  Note that np.nan is itself a legitimate complex128 value.

If the allocated array cannot hold the values from the block manager, and if the EA is not immutable, it will try a second strategy of allocating an array based on the dtype of the values of the blocks.  If the blocks hold complex numbers, the array will be properly sized.

This should close #54445.

- [x] closes #54445
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
I think the numpy failure is due to other CI/CD problems and not this PR.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
REF: Refactor using_pyarrow check for string tests
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: Strings
description: String extension data type and string data

Comments:
Thanks @phofl