BrandonFors/pandas-github-issues huggingface dataset:
CI: Test numpy 1.24 on 3.11 build
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration

Comments:
Why don't we try unpinning numpy on 3.11 instead? That build has no numba IIRC.
Hm did not consider this, thanks for pointing it out. Makes sense to me
cc @mroeschke this is ready for a review now, not sure if we want to leave all the ignores in...
> not sure if we want to leave all the ignores in

If there are any deprecation or future warnings those should be fixed
We are only filtering RuntimeWarnings now.
Were any of the RuntimeWarnings addressable, or were they specific to the data being passed?
All data dependent.
Thanks @phofl 
I think these tests are failing due to numpy being unpinned here

```
 FAILED pandas/tests/plotting/test_series.py::TestSeriesPlots::test_bar_log - AssertionError: numpy array are different

numpy array values are different (16.66667 %)
[left]:  [0.09999999999999999, 1.0, 10.0, 100.0, 1000.0, 10000.0]
[right]: [0.1, 1.0, 10.0, 100.0, 1000.0, 10000.0]
FAILED pandas/tests/plotting/frame/test_frame_subplots.py::TestDataFramePlotsSubplots::test_bar_log_no_subplots - AssertionError: numpy array are different

numpy array values are different (25.0 %)
[left]:  [0.09999999999999999, 1.0, 10.0, 100.0]
[right]: [0.1, 1.0, 10.0, 100.0]
FAILED pandas/tests/plotting/frame/test_frame_subplots.py::TestDataFramePlotsSubplots::test_bar_log_subplots - AssertionError: numpy array are different

numpy array values are different (16.66667 %)
[left]:  [0.09999999999999999, 1.0, 10.0, 100.0, 1000.0, 10000.0]
[right]: [0.1, 1.0, 10.0, 100.0, 1000.0, 10000.0]
```
Yikes this is weird, I xfailed them but this build started failing because the tests were passing again, so I removed it…
BUG: Series.interpolate with dt64/td64 raises
State: closed

- [x] closes #11312 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Thanks @jbrockmendel
REF: Groupby.quantile allow EA dispatch
State: closed

cc @jorisvandenbossche @mroeschke 

This is partially a proof of concept for something I think we need to do for most of the groupby reductions (more generally, just about anywhere we call Manager.grouped_reduce and friends)

Many groupby reductions(ish) have special handling to convert EAs to numpy, then pass them to our cython code, then convert back.  This would be pretty inefficient for the new pyarrow dtypes given that IIUC they have native implementations of some of the relevant methods.  (I'm also imagining a future in which dask/modin/etc implement distributed EAs or cudf has as GPUEA)

I think this will also make it easier to fix e.g. dt64tz and PeriodDtype cases which are currently broken.

This puts the whole thing in an EA method groupby_quantile and calls that in EA cases.  Lots of other reasonable ways this can be refactored; I'm open to suggestions.

Labels:
name: Groupby
name: quantile
description: quantile method

Comments:
@mroeschke any big-picture thoughts here and/or #51116?  i think something like this is inevitable, but am not married to this particular design pattern
My one concern is how much "groupby" implementation the EA needs to be aware of in the signature of a groupby specific EA method. In an ideal world it would be great if the groupby operation just dispatches to the EA's _reduce.

If this is needed/useful in the near term, it would be great if the implementation was "private" since how much groupby implementation the array needs to be aware of could change.
updated to privatize
> This would be pretty inefficient for the new pyarrow dtypes given that IIUC they have native implementations of some of the relevant methods

FWIW the groupby kernels (like `hash_sum`) are not actually exposed right now as is in pyarrow (they are only usable in context of the full query engine, not as standalone kernels)

A bit off-topic, but specifically for the pyarrow dtypes, I also think that we could actually use the support for masked arrays in our cython groupby algos. The only conversion step then is the bitmask to a bytemask, but which should be cheaper than converting to a numpy array (if there are nulls).
> specifically for the pyarrow dtypes, I also think that we could actually use the support for masked arrays in our cython groupby algos

+1.  i think something like a `_to_masked_array` and `_from_masked_array` would be useful for a bunch of these methods medium-term.
Closing, as I've convinced myself the cython quantile function is unecessary xref #51385
BLD: Drop universal2 wheels
State: closed

Numpy hasn't provided them anymore for a while, so let's drop them as well to cut down on compilation time and save space on PyPI.

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Build
description: Library building on various platforms
name: OS X
description: Related to Mac OS & hardware issues (M1)

Comments:
Thanks @lithomas1
CLN validate some Py_* Functions in C Extensions check for NULL
State: closed

xref #49756

@WillAyd is this what you had in mind?

In cases like

https://github.com/pandas-dev/pandas/blob/3a1a8b3c6bc94a3036e45d75adf33adc5b35c697/pandas/_libs/src/ujson/python/objToJSON.c#L902-L906

which return `void`, how should `NULL` be handled?

Labels:
name: Clean
name: Stale

Comments:
For functions that return `void` your hands are a bit tied. In that particular case `PyObject_Dir` should be setting the global python indicator, so somewhere along the lines you could do a `PyErr_Occurred()` check and see what happened. But that's thorny so I would just ignore for now.

Would be curious how that gets handled upstream in ultrajson. Seems like there's a much more natural way to serialize the built-in types than what we have here
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
thanks for your review

I probably won't have a chance to update this any time soon due to there being quite a few other tasks to take care of, so closing for now to clear the queue
BUG: to_datetime with out-of-bounds np.datetime64 and errors='ignore' doesn't return input
State: closed

- [ ] closes #50587  (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Datetime
description: Datetime data dtype

Comments:
I'm on board with this change, should probably make sure others agree
> I'm on board with this change, should probably make sure others agree

sure - @mroeschke any objections? Anyone I should ping who might be opinionated on this?
thanks! and yes, errors='coerce' with oob is covered, e.g.


https://github.com/pandas-dev/pandas/blob/4a61e6a3f8b0e5f2ebad098cc53eb4376927915a/pandas/tests/tools/test_to_datetime.py#L1089-L1095
Thanks @MarcoGorelli (failure unrelated)
nice!
DEPR: Move Numericindex._convert_slice_indexer & ._maybe_cast_slice_bound
State: closed

Moves`_convert_slice_indexer` & `_maybe_cast_slice_bound` from `NumericIndex`to `Index` in preparation to remove `NumericIndex` and include numpy int/uint/float64 in the base `Index`.

Each commit contains one method moving each, could be easier to view each method separately in the commit. 

xref #42717.

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Updated.
thx @topper-123
Min max added to reduction ops
State: closed

- [x] closes #50937 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: expressions
description: pd.eval, query

Comments:
Can someone from pandas team review this?
> Needs tests and a whatsnew note in `v2.0.0.rst`

Could you give some pointers on how can I add these tests and whatsnew
A unit test that checks the behavior of the related issue be added to `pandas/tests/frame/test_query_eval.py`.

An entry should be added to `doc/source/whatsnew/v2.0.0.rst` describing what was fixed
> A unit test that checks the behavior of the related issue be added to `pandas/tests/frame/test_query_eval.py`.
> 
> An entry should be added to `doc/source/whatsnew/v2.0.0.rst` describing what was fixed

Would the following test be okay?

```
class TestDataFrameQueryNumExprPython(TestDataFrameQueryNumExprPandas):

    ...

    def test_query_numexpr_with_min_and_max_columns(self):
        df = DataFrame({"min": [1, 2, 3], "max": [4, 5, 6]})
        regex_to_match = r"Variables in expression \"\(min\) == \(1\)\" overlap with builtins: \('min'\)"
        with pytest.raises(NumExprClobberingError, match=regex_to_match):
            df.query("min == 1")

        regex_to_match = r"Variables in expression \"\(max\) == \(1\)\" overlap with builtins: \('max'\)"
        with pytest.raises(NumExprClobberingError, match=regex_to_match):
            df.query("max == 1")

```
Yes but the tests can just be stand alone functions
> An entry should be added to `doc/source/whatsnew/v2.0.0.rst` describing what was fixed

Also, could you give me some more hints on what format I should follow to add changes descriptions in 
`doc/source/whatsnew/v2.0.0.rst`? Like in what section, what should be the phrasing format, etc.

Is this ok:

> Numeric
> ^^^^^^^
> - Bug in :meth:`query` with ``engine="numexpr"`` and column names are ``min`` or ``max``, does not work correctly (:issue:`50937`)
This will be easier to comment on if you commit the changes you suggested
> This will be easier to comment on if you commit the changes you suggested

Sure!
```
************* Module pandas.tests.frame.test_query_eval
pandas/tests/frame/test_query_eval.py:540:8: W0404: Reimport 'NumExprClobberingError' (imported line 6) (reimported)
```
Thanks @ShisuiUzumaki
REF: ArrowEA _data->_pa_array
State: closed

Totally subjective: I prefer a distinctive/informative name.  This makes it easy to e.g. grep for all the places where we access this attribute.

Labels:
name: Refactor
description: Internal refactoring of code
name: Arrow
description: pyarrow functionality

Comments:
@mroeschke if you're not on board we can close this, otherwise ill rebase
Curious, would this make it harder for `ArrowExtensionArray` to try to use masked array methods since it calls `self._data` a lot?
> Curious, would this make it harder for ArrowExtensionArray to try to use masked array methods since it calls self._data a lot?

It would if we did that, but they don't share any code...

Also the ._data attributes are different types, which i would think is exactly the kind of thing its nice to have clear names for
I was thinking of, for example, utilizing the base masked engine work in the short term to have better indexing support which I think utilizes `._data`.

But overall it's not that big of a deal, I am fine changing it to `_pa_array`
@mroeschke any objection to merging before more merge conflicts pile up?
> But overall it's not that big of a deal, I am fine changing it to `_pa_array`

yep. Go ahead and change if you feel it's an improvement.
no objections but also no enthusiasm.  im on the fence about self-merging.
Just fixed another merge conflict.  im planning to merge on green unless someone objects.
Thanks @jbrockmendel
BUG: ArrowExtensionArray.mode(dropna=False) not respecting NAs
State: closed

- [x] closes #50982 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Arrow
description: pyarrow functionality

Comments:
looks like this breaks the existing mode tests
Couple comments, neither deal-breakers.  Otherwise LGTM
@jorisvandenbossche can you take a look?  this calls a bunch of pyarrow stuff directly
Greenish
ENH: Add dtype_backend support to read_sql
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query
name: Arrow
description: pyarrow functionality

Comments:
Thanks @phofl
BUG: astype to pyarrow does not copy np array
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @mroeschke 

This should copy. You can't update the pyarrow array, but the parent array can get updated which propagates to the pyarrow array.

Labels:
name: Arrow
description: pyarrow functionality

Comments:
worth adding this to tm.shares_memory?
Raises a NotImplementedError (no arrow support)

Will have a look at this in a follow up
Thanks @phofl
BUG/ENH: groupby.quantile support non-nano
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @jbrockmendel
BUG: df_empty.convert_dtypes() with pyarrow backend
State: closed

- [x] closes #50970 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.

Doesn't need a whatsnew since this will be a new feature in 2.0


Labels:
name: Arrow
description: pyarrow functionality

Comments:
thx @mroeschke
DOC: update style.ipynb for 2.0
State: closed

This PR tries to frame Styler more as a general display output tool, by including the more generalist "formatting" functions at the start,

![Screenshot 2023-01-25 at 20 10 11](https://user-images.githubusercontent.com/24256554/214660804-c9ca098f-a419-4735-9136-381a06840879.png)
![Screenshot 2023-01-25 at 20 10 29](https://user-images.githubusercontent.com/24256554/214660828-7e7e52c3-4b73-44e8-8d65-67934ab6e123.png)


Labels:
name: Docs
name: Styler
description: conditional formatting using DataFrame.style

Comments:
```
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
df = pd.DataFrame({
    "strings": ["Adam", "Mike"],
    "ints": [1, 3],
    "floats": [1.123, 1000.23]
})
df.style \
  .format(precision=3, thousands=".", decimal=",") \
  .format_index(str.upper, axis=1) \
  .relabel_index(["row 1", "row 2"], axis=0)
------------------

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[1], line 1
----> 1 df = pd.DataFrame({
      2     "strings": ["Adam", "Mike"],
      3     "ints": [1, 3],
      4     "floats": [1.123, 1000.23]
      5 })
      6 df.style \
      7   .format(precision=3, thousands=".", decimal=",") \
      8   .format_index(str.upper, axis=1) \
      9   .relabel_index(["row 1", "row 2"], axis=0)

NameError: name 'pd' is not defined
NameError: name 'pd' is not defined

Notebooks DONE
```
Thanks @attack68
TST: Test ArrowExtensionArray with decimal types
State: closed

closes #34166

Labels:
name: Arrow
description: pyarrow functionality

Comments:
[XPASS(strict)] No pyarrow kernel for decimal128(4, 3)
Greenish. Merging.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 f5405b58e522f926e8fb6eef6c10ccb4ddf84a86
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #50964: TST: Test ArrowExtensionArray with decimal types'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-50964-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #50964 on branch 2.0.x (TST: Test ArrowExtensionArray with decimal types)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
Please accept my pull request
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
can you please experiment in your own repo? thanks 🙌 
sure, I will do experiment in my own repo
Enable pylint undefined-loop-variable warning
State: closed

- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Enables the Pylint warning `undefined-loop-variable`.  This contributes to (now-closed) #48855 and extends #49740.

Replaces several instances of
```
for x in my_list:
  foo(x)
bar(x)  # undefined-loop-variable
```
with
```
last_element = None
for x in my_list:
  last_element = x
  foo(x)
bar(last_element)  # no undefined-loop-variable
```

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @uzzell
DOC: Fix iteritems docs
State: closed

I know `iteritems()` is being removed in Pandas 1.6, but Pandas 1.5.4 should at least fix the docs so it properly points to the recommended alternative.

Labels:
name: Docs

Comments:
pls merge main
@jbrockmendel That's the thing - I can't merge into main because the docs for `iteritems()` was removed already in main, ahead of the 1.6 release. This fix only needs to go into the 1.5.x branch. (So that current and future users of the 1.5.x docs have a good link, assuming that there will be another 1.5.x release)
> @jbrockmendel That's the thing - I can't merge into main because the docs for `iteritems()` was removed already in main, ahead of the 1.6 release. This fix only needs to go into the 1.5.x branch. (So that current and future users of the 1.5.x docs have a good link, assuming that there will be another 1.5.x release)

Thanks @hintron for the PR. This indeed does present some complications considering our regular process.

This PR is opened against pandas-dev:main. If we do a 1.5.4 release than we could potentially consider a doc improvement.

I'm closing this but could you either open an issue describing the problem or re-submit the PR against pandas-dev:1.5.x. The later is not normal procedure so make sure to highlight this in the opening comment.