BrandonFors/pandas-github-issues huggingface dataset:
PERF: assert_frame_equal and assert_series_equal for frames/series with a MultiIndex
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.


```
import pandas as pd

N = 500

dates = pd.date_range("2000-01-01", periods=N)
strings = pd._testing.makeStringIndex(N)

mi = pd.MultiIndex.from_product([dates, strings])

df1 = pd.DataFrame({"a": 1}, index=mi)
df2 = df1.copy(deep=True)

%timeit pd.testing.assert_frame_equal(df1, df2)

# 4.27 s ± 245 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)    -> main
# 10 ms ± 415 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  -> PR
```

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Performance
description: Memory or execution speed performance

Comments:
Nice! Thanks @lukemanley 
nice!
REF: Remove un-used attribute-pinning in plotting
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

xref #54485

@tacaswell can you confirm that matplotlib doesn't use any of these internally?  Removing these doesn't break any of our tests locally, but a lot of our plotting tests are basically smoke-tests.

Labels:
name: Visualization
description: plotting

Comments:
I also believe that these are attributes that Pandas monkey-patches onto Matplotlib objects.

If you want to stash state related to unit conversion you can use the `Axis.units` attribute (best vias `Axis.set_units(...)`.

attn @ksunden
Thanks @jbrockmendel 
This commit appears to be causing an order of magnitude performance issue when plotting with a DatetimeIndex. See https://github.com/pandas-dev/pandas/issues/57035
ENH: Improve error message when constructing period with invalid freq
State: closed

xref #55792, #52064

Improved error message when constructing period with offsets frequencies deprecated for period such as "ME" or "YE-DEC".

the reason: we see the error message: `Invalid frequency: "..."` instead of the message: `for Period, please use "..." instead of "..."`

Labels:
name: Frequency
description: DateOffsets
name: Deprecate
description: Functionality to remove in pandas

Comments:
@MarcoGorelli, could you please take a look at this PR?
ASV: add FrameMixedDtypesOps
State: closed

- [ ] closes #31075
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Added mixed-dtype reduction asv


Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
Thanks @quangngd
ASV: DataFrame.__setitem__ performance with object dtype
State: closed

- [ ] closes #19299 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x]  Added new benchmark SetitemObjectDtype

Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
Thanks @quangngd
BUG: Set check_exact to true if dtype is int
State: closed

- [x] closes #55882 
- [x] [Tests added and passed]
- [x] All [code checks passed]
- [ ] Added [type annotations]
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
 @MarcoGorelli not sure where to add test cases for the change.
 
 Edit: Got it (pandas/tests/util/test_assert_frame_equal.py) 
Well I'm relieved that CI is still green 😄 Can you update the docs for this method to note that `rtol`, `atol` and `check_exact` don't take effect for int dtype?

Regarding the check - not totally sure about the check, will take a closer look next week
Sure. Updated the documentation. Added note under `check_exact`. Since we are setting it to True in case of int dtype. Also, documentation for `rtol` and `atol` states it is applicable only when check_exact is False so I assumed that it is sufficient to add note under `check_exact`
I think it should always be exact, except for floats?

You should also check for extensions arrays, and include a whatsnew

I've added a commit anyway, hope it's ok - do the changes look alright to you?
Yeah, make sense.

Some testcases are failing in some builds. Is that alright?
Looks like another bug was hiding because of the check_exact default 😄  https://github.com/pandas-dev/pandas/issues/56136

For now let's xfail the relevant test
Sure Marco
Thanks @parthi-siva and @MarcoGorelli 
> Looks like another bug was hiding because of the check_exact default 😄 #56136
> 
> For now let's xfail the relevant test

@jorisvandenbossche yeah.. there was another issues because of this
BUG: pivot_table with margins and numeric columns
State: closed

- [ ] closes #26568
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit)
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Thanks @quangngd
PERF: Cython 3.0 regression with frame constructor
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Handles the 5 `frame_ctor.FromDicts` regressions in #55179

There is only one use of this function:

https://github.com/pandas-dev/pandas/blob/b2d9ec17c52084ee2b629633c9119c01ea11d387/pandas/core/internals/construction.py#L627

and it's guaranteed to always have object dtype:

https://github.com/pandas-dev/pandas/blob/b2d9ec17c52084ee2b629633c9119c01ea11d387/pandas/core/internals/construction.py#L618-L619

ASVs on this branch:

```
| Change   | Before [b2d9ec17] <perf_cython3_frame_dict_constructor~1>   | After [951c5bfa] <perf_cython3_frame_dict_constructor>   |   Ratio | Benchmark (Parameter)                               |
|----------|-------------------------------------------------------------|----------------------------------------------------------|---------|-----------------------------------------------------|
| -        | 17.2±0.2ms                                                  | 15.3±0.3ms                                               |    0.89 | frame_ctor.FromDicts.time_nested_dict               |
| -        | 13.2±0.2ms                                                  | 11.7±0.3ms                                               |    0.89 | frame_ctor.FromDicts.time_nested_dict_index_columns |
| -        | 13.1±0.2ms                                                  | 11.3±0.3ms                                               |    0.86 | frame_ctor.FromDicts.time_nested_dict_index         |

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```

Labels:
name: Performance
description: Memory or execution speed performance
name: DataFrame
description: DataFrame data structure
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
Thanks @rhshadrach nice find
Fix for 55753 BUG: No kwargs in df.apply(raw=True) [  Backport to 2.1.x ]
State: closed

- [x] closes #55753
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
done, updated the whatsnew
Thanks @dannyi96 
I think we needed to move the whatsnew from 2.2.0 to 2.1.3, since there's an existing whatsnew already from the original PR that fixed this.

@dannyi96 Interested in doing another PR against main fixing this?
sure @lithomas1.
do you mean removing the related whatsnew in v2.2.0.rst since its already been added in v2.1.3.rst ( & hence not a new fix in 2.2.0) ?
You need to move it since your changes only applied to 2.1.x not main.
BUG: Keep original values when taking with a new fill value
State: closed

- [x] closes #55181
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.

`SparseArray`'s `take` method could mix up the two possible fill values: the array's fill value and the `fill_value` argument provided to `take`. In the case of a non-empty sparse array whose values were all its fill value, `take` would ignore the array's fill value and only use its `fill_value` argument, even for valid indices. The return value would be all `fill_value`, rather than a mix of the two fill values.


Labels:
name: Sparse
description: Sparse Data Type

Comments:
Thanks @rob-sil
Fixes #55884- Added else for invalid fill_method
State: closed

- [x] Closes #55884 
- [x] Tests added and passed
- [x] All code checks passed

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
@lithomas1 could you please review this ^^? Thanks a lot !
@mroeschke All suggestions were implemented. Thanks a lot for the guidance. This was very instructive !
Thanks @ziadk
TST: Group by callable test
State: closed

The bug is fixed. Add the tests

- [ ] closes #22257
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).\


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @quangngd
DOC: Clarify DataFrame.dropna lib.no_default kwarg behavior
State: closed

- [x] closes #55848

Changes
<img width="745" alt="Screenshot 2023-11-11 at 9 42 37 PM" src="https://github.com/pandas-dev/pandas/assets/25352646/44286bca-31d8-45dd-a47f-a4697e53098f">

To
<img width="917" alt="Screenshot 2023-11-11 at 9 42 12 PM" src="https://github.com/pandas-dev/pandas/assets/25352646/e53c7b6a-e9a6-48e7-8a3a-dfb518d39990">

by adding one-line function declaration to doc-string per sphinx default autodoc_docstring_signature behavior.

Labels:

Comments:
@kamilkrukowski I don't think the changes in this PR make sense. Not sure if the solution you want to implement is as easy as you think. I agree with your points in the issue, but as Thomas said, we want to keep the current behavior, even if unfortunately exposes some confusing internal details to the users.

If you want to change the header rendered by Sphinx you will have to do it with a Sphinx plugin. Adding or changing things in the docstring won't change the signature. Only changing the Python code signature, which we don't want.

I'll close this issue, happy to follow up if I misunderstood what you are doing, or you want to discuss further. But I think what you want to achieve is too complex to be worth. So, unfortunately users will have to live with the nodefault argument in the documentation.
ASV: clean/simplify setup methods
State: closed

Trying to make some of the setup methods a bit more efficient. Might(?) save around a minute total on the ASV job. 

e.g.

```
import time
from itertools import product
from asv_bench.benchmarks.series_methods import NanOps

param_sets = list(product(*NanOps.params))

t = time.time()
for params in param_sets:
    try:
        NanOps().setup(*params)
    except NotImplementedError:
        pass
print(time.time() - t)

# 11.87  -> main
# 1.90   -> PR
```


Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
Thanks @lukemanley
Create create-issue.yml
State: closed

This is a GIthub Action that creates a new issue (and delete the old one) reporting all the errors and label and Docs and Good First Issue for a broken link.

- [x] closes #45409 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Hi @pmeier, how to proceed with the  [3rd part](https://github.com/pandas-dev/pandas/issues/45409#issue-1105272826) ? The 2nd part is merged.

Almost every time the `linkcheck file` is run there is a new broken link found. Is there a way to automatically append that link into the `ignore list` ?

pmeier Is there any way to Dm you for the PR? I had tried to tag you in the merged PR , but no luck. Do we get notifications in merged or closed PRs ?
> Almost every time the linkcheck file is run there is a new broken link found. Is there a way to automatically append that link into the ignore list ?

No, and that is not the point. When this workflow is running on a schedule and finds broken links, a new issue should be created containing all the broken links so a human can come and fix them. 

Just adding everything to the ignore list is the opposite of what we want. Ideally, we don't want to have an ignore list at all. This is likely not possible, because some links like we have seen for `http://` will never work so they have be ignored. But the ignore list should be as short as possible.

So if you are stuck on this PR, you can also work on reducing the ignore list. 
Thanks for the help with this @ggold7046. The issue you are trying to address is not trivial, and the changes in this PR don't seem to be addressing the issue. I would suggest that you work on an issue that you fully understand. I'll close here, feel free to open a PR once you're sure you understand the problem and you have a reasonable solution. Thank you!
> So if you are stuck on this PR, you can also work on reducing the ignore list.

@pmeier,  I would like to work on the last part of [this](https://github.com/pandas-dev/pandas/issues/45409#issue-1105272826) from where I started. 

> 3. If errors are found, have the GIthub Action create a new issue (and delete the old one) reporting all the errors and label and Docs and Good First Issue

I would like to work on this one.
> this PR don't seem to be addressing the issue.

Hi @datapythonista , It was a PR work in progress.
Fix off-by-one in aggregations
State: closed

ref https://github.com/pandas-dev/pandas/issues/55713#issuecomment-1806580402

I'm not super familiar with this function so maybe there is a more intelligent way of doing this. But the segfault occurs when self.window_size == 0 because of these lines:

```python
end = np.arange(1 + offset, num_values + 1 + offset, step, dtype="int64")
start = end - self.window_size
```

In the segfaulting example, `num_values` is 20 for an array with 20 elements. `end` ends up becoming 1-20 as does start with the window_size of 0. Later code then tries to index the 20th index in array, hence the address violation

Labels:
name: Window
description: rolling, ewma, expanding
name: Segfault
description: Non-Recoverable Error

Comments:
Thanks @WillAyd (for fixing my math mistake 😅 )
REF: less state in scatterplot
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: Visualization
description: plotting

Comments:
Thanks @jbrockmendel
Fix Cython 3.0 regression with time_loc_dups
State: closed

re https://github.com/pandas-dev/pandas/pull/55179

From discussion in https://github.com/cython/cython/issues/1807#issuecomment-1802656280 it looks like Cython  prior to 3.0 would always use the sequence protocol for indexing with an integral value. However, Python prefers the object protocol first if available, and Cython switched to match that logic with 3.0

NumPy arrays implement both the sequence and the mapping protocol. In cases where we have untyped arrays that fall back to Python calls we will see a performance regression since this will now route through the mapping space

The changes in this PR are not meant to be an exhaustive review of the codebase, rather just a quick POC to reset the time_loc_dups benchmark


Labels:
name: Performance
description: Memory or execution speed performance
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
LGTM, bummer that its necessary though.
Thanks @WillAyd
Backport PR #55907 on branch 2.1.x (DOC: Add release date for 2.1.3)
State: closed

Backport PR #55907: DOC: Add release date for 2.1.3

Labels:
name: Docs

Comments:
The doc build is expected to fail here (being fixed by https://github.com/pandas-dev/pandas/pull/55912), so will merge here
WIP set full column if using self.index in loc
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

closes https://github.com/pandas-dev/pandas/issues/55791

Labels:

Comments:
nvm this is a bad idea, `df.loc[df.index]` is currently different from `df.loc[:]` and I don't want to risk breakage
REF: avoid altering self.data in get_xticks
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: Visualization
description: plotting

Comments:
Thanks @jbrockmendel
REF: avoid statefulness in 'color' args
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: Visualization
description: plotting

Comments:
Thanks @jbrockmendel