BrandonFors/pandas-github-issues huggingface dataset:
PERF (CoW): optimize equality check in groupby is_in_obj
State: closed

See discussion in https://github.com/pandas-dev/pandas/pull/49450

TODO:

- [ ] Probably need to add more tests (ensure full helper is covered)

Labels:
name: Stale

Comments:
cc @jbrockmendel are you aware of something similar that already exists in the code base?
> are you aware of something similar that already exists in the code base?

nope, will be nice to have
> Can of course easily check that the strides are equal. But is that enough guarantee?

IIRC I discussed something similar with @seberg a while back.  I don't remember what he said, other than suggesting an approach that didn't involve checking memory.
I remember a brief discussion at some point, but not really what I suggested or the details...

If all you want is to check identity, then you can check the starting pointer and the stride(s).  `np.shares_memory(arr1[:1], arr2[:1])` is hard to confuse (possible in NumPy, but you have to construct it with will).

Maybe I noted that overlap checking is difficult (not sure how hard it is exactly in 1-D), but since you are interested in identity checking that doesn't matter.
I guess I always imagined that CoW implementations would have a `view.base` and `weakref`'s back to keep track of what is shared (so the parent knows all its views), but I don't actually know anything about CoW implementations, so :)...
> I guess I always imagined that CoW implementations would have a `view.base` and `weakref`'s back to keep track of what is shared (so the parent knows all its views),

That's exactly what we do in our CoW implementation in the internals, but, here it's some special case in a groupby code path where we previously relied on actual (python object level) identity, which no longer works with CoW enabled (getting a column twice no longer gives python-identical objects). So I was trying to find some way to see if two objects had exactly the same memory as alternative check (although not exactly equivalent, as an actual view would share the same memory but not be identical).

But now you say this, I actually didn't think of that we could use this base/refs information ... Thanks for the insight! :)
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Closing in favor of https://github.com/pandas-dev/pandas/pull/51442, which essentially does the last suggestion here (using the ref information to know if they are views of each other)
TST: mark copy_view test_switch_options test as single_cpu
State: closed

While backporting https://github.com/pandas-dev/pandas/pull/49771 in https://github.com/pandas-dev/pandas/pull/50128, I had to mark the test to not run in parallel to get it passing. This was not needed on the main branch, but since this can depend on the exact order and presence of other tests, seems safer to apply this to the main branch as well.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thx @jorisvandenbossche
CI fix failing python-dev job
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

without `--pre`, it wasn't taking the latest nightly

Labels:
name: CI
description: Continuous Integration

Comments:
thx @MarcoGorelli
REF: stricter typing, better naming in parsing.pyx
State: closed

Trying to move towards having fewer different string parsing paths.

Labels:
name: Datetime
description: Datetime data dtype
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @jbrockmendel 
Because I couldn't find it in the changelog (probably because it's supposed to be an internal function no one should have actually used):
This PR removed the `parse_time_string` function from `pandas.core.tools.datetimes`

We were using it in nextstrain/augur's parse to get information about the datetime resolution. Currently seeing how to migrate to pandas v2 given the removal.
BUG: .max() gives wrong answer for non-nanosecond datetimeindex
State: closed

- [ ] closes #50716 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Does this need a whatsnew? I wouldn't have thought so, as this wasn't available in 1.5.x

Labels:
name: Bug
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
thanks - merging then as this is quite simple and unblocks rewriting tests for #50656
ENH: support any/all for pyarrow numeric and duration dtypes
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Cuts the runtime for `pytest pandas/tests/extension/test_arrow.py::TestBaseBooleanReduce::test_reduce_series` from 39.8s to 2.3s.

Labels:
name: Arrow
description: pyarrow functionality

Comments:
updated per requests + green
Thanks @jbrockmendel
PERF: Fix reference leak in read_hdf
State: closed

- [ ] closes #37441  (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

With my really hacky benchmark, I get,

```

       before           after         ratio
     [bb18847e]       [b5ddc810]
     <main>           <fix-hdf-memusage>
-            6.6M                0     0.00  io.hdf.HDF.mem_read_hdf_index('table')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```


Labels:
name: Performance
description: Memory or execution speed performance
name: IO HDF5
description: read_hdf, HDFStore

Comments:
Thanks @lithomas1
BUG: Fix style.background_gradient() for nullable dtype (ex: Int64) series with pd.NA values
State: closed

- [x] closes #50712
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Styler
description: conditional formatting using DataFrame.style

Comments:
@attack68 Ok cool, I just re-merged master so hopefully it all works this time. If you look at one of the previous test runs, everything looks ok minus one windows test that seems somewhat flaky

![image](https://user-images.githubusercontent.com/2521996/212335587-510bf4ba-cdc9-4f7c-9115-f68401dddd5b.png)

Since then all I have done is re-merge master
All comments addressed. 

@mroeschke @attack68 Thanks for the review
Thanks @dhendry
DOC: added installation instructions for nightly build
State: closed

- [x] closes #50703
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks for the PR.
Can you also add a note here?
https://github.com/pandas-dev/pandas/blob/main/.github/ISSUE_TEMPLATE/bug_report.yaml
(Next to "I have confirmed this bug exists on the main branch of pandas")
Thanks @lpizzinidev
Backport PR #49615 on branch 1.5.x (REGR: Better warning in pivot_table when dropping nuisance columns)
State: closed

… nuisance columns

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Warnings
description: Warnings that appear or should be added to pandas
name: Nuisance Columns
description: Identifying/Dropping nuisance columns in reductions, groupby.add, DataFrame.apply

Comments:
I think this was taken care of by #49653 but label was not removed
yeah just saw that 😄 
We should be more strict in linking the backport pr to the original pr, this is really hard to see like this
Backport PR #49386 on branch 1.5.x (DOC: add name parameter to the IntervalIndex for #48911)
State: closed

…#48911

Backport PR #49386

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
@MarcoGorelli do you need to move release note also?
I'm not sure the original PR should have had a release note TBH if it was just a documentation change, I'd be inclined to just remove it from there
> I'd be inclined to just remove it from there

sgtm
CI capture error in numpy dev, fix python-dev build
State: closed

- [ ] closes #50695 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

the macos job fails during the installation phase: https://github.com/pandas-dev/pandas/issues/50699


Labels:
name: CI
description: Continuous Integration

Comments:
sure, thanks!
@meeseeksdev please backport to 1.5.x
CI: Add regular 3.11 pipeline
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration

Comments:
cc @MarcoGorelli 

Looks like using mamba for the env fixes the installation issue, but we have some weird segfault on the 3.11 single cpu build
`--extra-index-url` instead of `-i` seems to have fixed it too https://github.com/pandas-dev/pandas/pull/50698

if that works, would this one still be necessary?
This one takes precedence over the dev build. It's kinda unrelated but after seeing the failures today I remembered that I wanted to look into it anyway.

The dev build only tests with our required dependencies while this one enables a full test-run on 3.11 (except numba since it's not compatible with 3.11 yet). This would give us the same test coverage as the other supported python versions.

Have to get the ci passing before that though.
pytables is not yet available for 3.11 on conda-forge.
Last build was green on 3.11, merging main to get up to date. @lithomas1 would you mind having a look?
Merging to get our stuff tested properly on 3.11
TST: CoW with df.isetitem()
State: closed

xref https://github.com/pandas-dev/pandas/issues/49473

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Copy / view semantics

Comments:
thx @lithomas1
TST: incorrect pyarrow xfails
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Arrow
description: pyarrow functionality

Comments:
updated + green
Thanks @jbrockmendel
BUG: ArrowDtype.construct_from_string round-trip
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Arrow
description: pyarrow functionality

Comments:
Might make sense to feature request this to Arrow to support parsing tz's in `type_for_alias` so we don't have to roll our own?
> Might make sense to feature request this to Arrow to support parsing tz's in type_for_alias so we don't have to roll our own?

I'll do that separately, sure.
issue has been opened upstream + green
@mroeschke gentle ping on pyarrow-xfail PRs
CI timeouts unrelated, merging.
TST: simplify pyarrow tests, make mode work with temporal dtypes
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Arrow
description: pyarrow functionality

Comments:
If I am understanding https://github.com/apache/arrow/pull/33685, then in pyarrow 11 we won't have to do the the int64 workaround?
> If I am understanding https://github.com/apache/arrow/pull/33685, then in pyarrow 11 we won't have to do the the int64 workaround?

I don't know if that will affect mode directly.
updated to use .cast instead of .astype
Thanks @jbrockmendel
BUG: Series.quantile emitting RuntimeWarning for all NA case
State: closed

- [x] closes #50681 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I am not totally sure that we should backport since technically this also fixes a bug. 

Edit: Could probably avoid the bug fix I guess

Labels:
name: Bug
name: quantile
description: quantile method
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
No the new behavior is correct (long term). So should be good to go 
Thanks @phofl
BUG: pivot_table with nested elements and numpy 1.24
State: closed

- [ ] closes #50342 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I think we have still pinned numpy in our 1.5.x branch but confirmed this still works locally

```
% conda list numpy
# packages in environment at /opt/miniconda3/envs/pandas-dev:
#
# Name                    Version                   Build  Channel
numpy                     1.24.1           py38hc2f29e8_0    conda-forge
numpydoc                  1.5.0              pyhd8ed1ab_0    conda-forge

% pytest pandas/tests/reshape/test_pivot.py -k test_pivot_table_with_mixed_nested_tuples
================================================================================== test session starts ===================================================================================
platform darwin -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
rootdir: , configfile: pyproject.toml
plugins: asyncio-0.20.2, cython-0.2.0, hypothesis-6.58.1, xdist-3.0.2, cov-4.0.0, anyio-3.6.2
asyncio: mode=strict
collected 539 items / 538 deselected / 1 selected

pandas/tests/reshape/test_pivot.py .

------------------------------------------------------- generated xml file: /test-data.xml --------------------------------------------------------
================================================================================== slowest 30 durations ==================================================================================
0.01s call     pandas/tests/reshape/test_pivot.py::TestPivotTable::test_pivot_table_with_mixed_nested_tuples

(2 durations < 0.005s hidden.  Use -vv to show these durations.)
=========================================================================== 1 passed, 538 deselected in 0.48s ============================================================================
```

EDIT: Numpy is no longer pinned on main so this should run in the CI

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Decided not to check the numpy version per se since in prior numpy versions this was raising a DeprecationWarning internally