BrandonFors/pandas-github-issues huggingface dataset:
BUG: fix setitem with enlargment with pyarrow Scalar
State: closed

Potentially closes #52235

This updates `lib.is_scalar` to recognize pyarrow.Scalar objects. 

Further, it updates a very specific code path for setitem with enlargement with a scalar to preserve the arrow extension dtype (#52235). There are many other places where specific handling would need to be added for pyarrow Scalars for generally supporting them.

- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale
name: Arrow
description: pyarrow functionality

Comments:
In #27462 you suggested a method on the EADtype/EA to check for a recognized scalar (might be similar to _recognized_scalars on our datetimelike EAs).  I think this might be the right approach long-term. (which isn't mutually exclusive with this PR short term)

I went down a rabbit-hole looking at issues regarding is_scalar, have a half-written Deep Dive post about it.  For the time being I think we should hold off on updating is_scalar since it is public, do a targeted fix in the indexing code.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Looks like this PR has gone stale. Closing to clear the queue but feel free to reopen when you have time
PDEP6 implementation pt 3: block.where
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
this one causes errors for
```python
data1 = [[1, 2], [3, 4]]
data2 = [[0.5, 6], [7, 8]]
df1 = DataFrame(np.array(data1))
df2 = DataFrame(np.array(data2))
df = concat([df1, df2], axis=1)
df[df>2]
```
, I'll dig deeper but it might be better to leave it out (at least for now)
COMPAT: Use actual isinstance check for pyarrow.Array instead of hasattr(.. 'type') duck typing
State: closed

PR for the suggestion I made at https://github.com/pandas-dev/pandas/pull/52076/files#r1160653924


Labels:
name: Arrow
description: pyarrow functionality

Comments:
seems reasonable
In https://github.com/pandas-dev/pandas/pull/52833 I need a similar check, but in cython. So if we do that, we can probably only define it in cython instead of in `compat/pyarrow.py`
Moved it to cython given that we will likely need pyarrow in lib.pyx anyway (eg needed for https://github.com/pandas-dev/pandas/pull/52833 as well)
Can you be more specific? Which other specific checks are you thinking about? (beyond one of the main pyarrow classes)
Possibly pyarrow Table and Scalars too. There might be a use case for checking for specific Scalar or Array subclasses too?
I was referring to checks like, checking for an array/scalar and afterwards checking for a specific pyarrow type (int, ...). This would require helper functions for all checks we might want
> I was referring to checks like, checking for an array/scalar and afterwards checking for a specific pyarrow type (int, ...). This would require helper functions for all checks we might want

IIUC most of these would be relevant when we already have an ArrowExtensionArray and want to check if a scalar can be held in an array of that dtype, e.g. for setitem-with-expansion.  Most of our EAs have something like this, either _validate_scalar or _validate_setitem_value.  I think the way forward is to standardize this, so the relevant check would go on ArrowExtensionArray and we wouldn't need to worry about these helpers
@phofl OK to move forward here (pending green)?
Yep
@jorisvandenbossche can you merge main and ping on green
@jbrockmendel mypy is not happy with my attempt to add a type guard:

```
mypy.....................................................................Failed
- hook id: mypy
- duration: 132.62s
- exit code: 1

pandas/_libs/lib.pyi:46: error: Variable "pandas._typing.ArrowArrayTypeGuard" is not valid as a type  [valid-type]
pandas/_libs/lib.pyi:46: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
Found 1 error in 1 file (checked 1417 source files)
```
(and I can't directly seem to fix it)

OK with leaving the typing part out for a follow-up PR? (I can directly open one with what I tried here, so we can discuss the best approach with the typing experts)

> OK with leaving the typing part out for a follow-up PR? (I can directly open one with what I tried here, so we can discuss the best approach with the typing experts)

sounds good
thanks @jorisvandenbossche
CLN: internals.managers._form_blocks
State: closed

A small clean-up.

I ran `asv continuous -f 1.1 upstream/main HEAD -b frame_ctor` to see if this gave an performance effect, and got `"BENCHMARKS NOT SIGNIFICANTLY CHANGED."`

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
Thanks @topper-123
BUG: interchange bitmasks not supported in interchange/from_dataframe.py
State: closed

- [x] closes #49888 (Replace xxxx with the GitHub issue number)
- [x] closes #52845
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This would solve all the "interchange to pandas" issues I'm aware of, so if we could get it in we could really take the interchange protocol for a ride

Labels:
name: Interchange
description: Dataframe Interchange Protocol

Comments:
Thanks @MarcoGorelli
BUG: Non unitless np NaT arithmetic with non-nano
State: closed

- [ ] closes #52295 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
thx @mroeschke 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 eb9a3e81a946d5921e4458632e9632239ac8e8d1
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #52821: BUG: Non unitless np NaT arithmetic with non-nano'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-52821-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #52821 on branch 2.0.x (BUG: Non unitless np NaT arithmetic with non-nano)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BUG: DatetimeIndexResamplerGroupby as_index interaction with .agg()
State: closed

- [x] closes issue #[52397](https://github.com/pandas-dev/pandas/issues/52397) 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Groupby
name: Stale

Comments:
Hey @rhshadrach, can I ask for a second review ?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Apologies @szczekulskij - this fell off my radar. Are you interested in continuing?
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
[TST] BUG: grouping with categorical interval columns
State: closed

- [x] closes https://github.com/pandas-dev/pandas/issues/34164
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Labels:
name: Groupby
name: Categorical
description: Categorical Data Type
name: Interval
description: Interval data type
name: Needs Tests
description: Unit test(s) needed to prevent regressions

Comments:
thx @PrimeF
CLN: unify NumpyBlock, ObjectBlock, and NumericBlock
State: closed

This PR unifies `NumpyBlock`, `ObjectBlock`, and `NumericBlock` into `NumpyBlock`.

xref #52413

This is part of my effort to make it easier for pandas to support new-style dtypes like the variable width string dtype I'm working on that will hopefully replace usages of object string arrays in pandas (#47884). New-style dtypes have `dtype.kind` set to a null character, so using `dtype.kind` to discriminate between numpy dtypes as `pandas.core.internals.blocks.get_block_type` currently does makes it more difficult to support new-style dtypes, as I'd need to add additional checks there that will hurt performance in a hot code path. @jbrockmendel suggested this approach instead.

It wasn't clear to me what the backward compatibility guarantees are for `pandas.core.internals`. I think adding an `ObjectBlock` and `NumericBlock` that are (deprecated?) aliases to `NumpyBlock` will be sufficient to maintain back compat if that's desired.

As a bonus, it's no longer necessary to check `dtype.kind` twice in the block creation fast path - I moved the second check into the new `NumpyBlock.is_numeric` cached property - so I think this may be a tiny bit faster for benchmarks sensitive to block creation overhead.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
> It wasn't clear to me what the backward compatibility guarantees are for pandas.core.internals. I think adding an ObjectBlock and NumericBlock that are (deprecated?) aliases to NumpyBlock will be sufficient to maintain back compat if that's desired.

Its complicated.  We've landed on calling pandas.core.internals.api "pseudo-public" and acknowledging that some packages (pyarrow) access them even though they're not _really_ public.  Last time we removed a Block subclass there were complaints and we ended up reverting it and changing the removal to a deprecation (which was enforced in 2.0, so now it is gone).
OK, I’ll go ahead and add deprecated aliases. Thanks for the context and the prior art pointer. Will hopefully be pushing an update today that fixes the tests and addresses the comments.
Is there anything I can do on my end to move this forward?
Rebased to fix a conflict.
Would appreciate another round of review. The test failure is unrelated.
LGTM @ngoldbaum can you merge main one more time to be on the safe side
Nice work @ngoldbaum
Add test for pyarrow date32 repr
State: closed

- [ ] closes #48238
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Needs Tests
description: Unit test(s) needed to prevent regressions
name: Arrow
description: pyarrow functionality
name: Sprints
description: Sprint Pull Requests

Comments:
thx @gab50000
DOC: Adding examples for dtype_backend="pyarrow" for read_json()
State: closed

Added an example for dtype_backend="pyarrow" for read_json()
- [ ] closes https://github.com/noatamir/pyladies-workshop/issues/2
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: IO JSON
description: read_json, to_json, json_normalize
name: Arrow
description: pyarrow functionality
name: Sprints
description: Sprint Pull Requests

Comments:
@phofl, @noatamir, @jorisvandenbossche and @marcogorelli
thx @Dharni0607 
Thanks again Dharani!
TST: test loc index df with a MultiIndex and  constituent IntervalIndex
State: closed

- [x ] closes [#25298](https://github.com/pandas-dev/pandas/issues/25298)
- [ x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

@jorisvandenbossche or @noatamir

Labels:
name: MultiIndex
name: Interval
description: Interval data type
name: Stale
name: Sprints
description: Sprint Pull Requests

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Added test to confrim categorical perservation with merge
State: closed

- [ ] closes #37480 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Categorical
description: Categorical Data Type
name: Needs Tests
description: Unit test(s) needed to prevent regressions
name: Stale
name: Sprints
description: Sprint Pull Requests

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Issue 21932: Added Regression Test for index of pivot_table on empty table
State: closed

- [x] closes #21932 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature


@phofl, @noatamir, @jorisvandenbossche and @marcogorelli 



Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Needs Tests
description: Unit test(s) needed to prevent regressions
name: Sprints
description: Sprint Pull Requests

Comments:
thx @MirijaH
bug fix on issue 52799
State: closed

- [x] closes #52799 (Replace 52799 with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: interchange.from_dataframe doesn't work with large_string
State: closed

- [ ] closes #52795 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Interchange
description: Dataframe Interchange Protocol

Comments:
The change looks good.

One related thing to note: it might be good to test the full roundtrip, i.e. the conversion from pandas to pyarrow as well (or at least have a test that checks what we create for buffers in pandas' `__dataframe__`, to ensure that the type (u vs U) matches the offsets' dtype)
thanks @jorisvandenbossche , have updated and added a test - would appreciate it if we could get this in to encourage more testing of "from interchange object to pandas" scenarios in real life
It's good that the roundtrip works. Now, that only works because pyarrow actually checks the bitwidth of the offsets in case the type string is "u" (and not just assumes it will be int32 offsets). 
It's probably still good to make the pandas implementation compliant with the spec (use int32, or keep using int64 offsets but use "U" instead of "u" in the dtype specification)
thanks for explaining, noted!
Changed as requested in #52716 and added test for checking this issue…
State: closed

[] Changed as requested in #52716 and added test for checking this issue #52716

… #52749

- [ ] closes #52716 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: IO Excel
description: read_excel, to_excel

Comments:
Thanks @NikolayBoev
ENH: Support ArrowDtype in interchange Column.dtype
State: closed

- [x] xref #52323 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Arrow
description: pyarrow functionality
name: Interchange
description: Dataframe Interchange Protocol

Comments:
Going to merge this in. Can address any followups if needed