BrandonFors/pandas-github-issues huggingface dataset:
BUG: MultiIndex.get_indexer not matching nan values
State: closed

- [x] closes #37222 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This was bugging me for quite some time.

The previous solution did not provide a way to differentiate between matching missing values (e.g. NaN and NaN) and values from other that were missing in self. With shifting NaNs to 1 and values from other that are not in self to 0, we can calculate the correct result.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: MultiIndex

Comments:
Nice. Does this close all the issues linked in #48877? Are any of those tests worth adding?
It should, and yes. I'd like to merge the tests of your pr after this is through if you are ok with that
Thanks @phofl
BUG: pivot_table with margins=T raises when results in empty df
State: closed

- [x] closes #49240 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.0.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Thanks @debnathshoham
CI: maybe fix docs build
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thx @MarcoGorelli
BUG: Series(mixed_tz_datetime_objs, dtype=dt64tz)
State: closed

- [x] closes #32581 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Side-notes:
1) This also means we allow pretty much any mix of types e.g. `[Timestamp, pydatetime, int, float, pd.NaT, str]`
2) The implementation here is super-naive. It can be optimized if it becomes an issue.  But given the mess that array_to_datetime currently is, i'm happy erring on the side of simplicty.

Labels:
name: Datetime
description: Datetime data dtype
name: Timezones
description: Timezone data dtype
name: Series
description: Series data structure

Comments:
superceded by #49454
DEPR: Enforce disallowed merging scenarios
State: closed

Introduced in https://github.com/pandas-dev/pandas/pull/34810, https://github.com/pandas-dev/pandas/pull/40993, https://github.com/pandas-dev/pandas/pull/40991


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
lgtm excluding failures
Merging. Appears failure is unrelated
UPGRADE: Autoupdate pre-commit config
State: closed

closes #49406

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Doing all the required changes
Sir i did the changes but after doing some code to different lines ```E501 line too long (90 > 88 characters)``` the same errors came again but the chabges above were fixed
sir here
 ```
cdef inline void add_var(float64_t val, float64_t *nobs, float64_t *mean_x,
                         float64_t *ssqdm_x, float64_t *compensation,
                         int64_t *num_consecutive_same_value, float64_t *prev_value) nogil:
```
the line is too long and if i take and place nogil to next line it will give the errors as before so what has to be done
how about
```
cdef inline void add_var(
    float64_t val,
    float64_t *nobs,
    float64_t *mean_x,
    float64_t *ssqdm_x,
    float64_t *compensation,
    int64_t *num_consecutive_same_value,
    float64_t *prev_value
) nogil:
```
?
> 

trying it
It works both the tests have been passed
 Sir now what has to be done because the checks are failing
you still have invalid syntax, check the error messages
Sir the same errors came as before and now not getting the reason also of why they came
Thanks @ramvikrams for having started this - I've added a commit to clean the remaining things up
> Thanks @ramvikrams for having started this - I've added a commit to clean the remaining things up

thanks sir for your help all along
> Looks good to me, thanks @ramvikrams

Thanks sir for your help


<img src="https://media4.giphy.com/media/2OHYMJGnGoiVjVg55P/giphy.gif"/>
Mind addressing the merge conflict @ramvikrams 
> Mind addressing the merge conflict @ramvikrams

yes sir
@MarcoGorelli Sir can you please take a look at the merge conflicts I cannot get it
There were some unrelated CI failures, but they've been fixed upstream - hopefully this should now be green and ready to merge
>  hopefully this should now be green and ready to merge

Hoping for the same

Curious: what does #94906 mean in the PR? There aren't that many issues here...
probably a typo, thanks
enable pylint wildcard-import
State: closed

Issue #48855. This PR enables pylint type "W" warning: `wildcard-import`.
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Hi. During CI occurs one Error
> intersphinx inventory 'https://docs.scipy.org/doc/scipy/objects.inv' not fetchable due to <class 'requests.exceptions.ConnectionError'>: HTTPSConnectionPool(host='docs.scipy.org', port=443): Max retries exceeded with url: /doc/scipy/objects.inv (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f084715d220>: Failed to establish a new connection: [Errno 111] Connection refused'))

It doesn't look related to my changes.
CLN: assorted; silence test warnings
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Clean

Comments:
rebased+green
Thanks @jbrockmendel
enable pylint unused-wildcard-import
State: closed

Issue #48855. This PR enables pylint type "W" warning: `unused-wildcard-import`.
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Hi. During CI occurs one Error

> intersphinx inventory 'https://docs.scipy.org/doc/scipy/objects.inv' not fetchable due to <class 'requests.exceptions.ConnectionError'>: HTTPSConnectionPool(host='docs.scipy.org', port=443): Max retries exceeded with url: /doc/scipy/objects.inv (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff2721cf7c0>: Failed to establish a new connection: [Errno 111] Connection refused'))


It doesn't look related to my changes.
CLN: test_nanops.py
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Avoid `setup_method` being unnecessarily called on tests that don't require all the DataFrame/array objects that are being created

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
Hopefully addressed all the comments and green
thx @mroeschke
CLN: test_expressions.py
State: closed

- [x] closes #40497 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean
name: expressions
description: pd.eval, query

Comments:
thx @mroeschke
ENH: Add masked engine
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

```
       before           after         ratio
     [626b6510]       [efde9dd0]
-        270±10μs          148±3μs     0.55  indexing.NumericMaskedIndexing.time_get_indexer('UInt64', True)
-         210±5μs          106±1μs     0.51  indexing.NumericMaskedIndexing.time_get_indexer('Int64', True)
-        306±50μs          150±3μs     0.49  indexing.NumericMaskedIndexing.time_get_indexer('UInt64', False)
-       340±100μs          158±3μs     0.46  indexing.NumericMaskedIndexing.time_get_indexer_dups('Float64', False)
-        237±20μs        107±0.6μs     0.45  indexing.NumericMaskedIndexing.time_get_indexer('Int64', False)
-        360±70μs          157±2μs     0.44  indexing.NumericMaskedIndexing.time_get_indexer_dups('Float64', True)
-       369±100μs          155±3μs     0.42  indexing.NumericMaskedIndexing.time_get_indexer('Float64', False)
-        330±80μs          137±1μs     0.42  indexing.NumericMaskedIndexing.time_get_indexer_dups('UInt64', True)
-        335±50μs          140±2μs     0.42  indexing.NumericMaskedIndexing.time_get_indexer_dups('UInt64', False)
-       373±200μs          155±3μs     0.42  indexing.NumericMaskedIndexing.time_get_indexer('Float64', True)
-        270±30μs          106±1μs     0.39  indexing.NumericMaskedIndexing.time_get_indexer_dups('Int64', False)
-        310±60μs          110±1μs     0.36  indexing.NumericMaskedIndexing.time_get_indexer_dups('Int64', True)
```

Lets see if ci passes


Labels:
name: Enhancement
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
im not wild about the complexity this adds to the non-cython indexing code.  wondering if we can push more of it up into the cython code.

longer-term anticipating the need for a pyarrow-engine.  is that something we can expect pyarrow to implement?
@jbrockmendel so you would prefer unpacking data and mask on the cython level in the engine specific code?
> @jbrockmendel so you would prefer unpacking data and mask on the cython level in the engine specific code?

I think so, yes.  but i kinda think it'll be messy regardless so either way is no big deal

This is less ugly now :)
cc @jbrockmendel Could you have another look?
Added
Thanks @phofl 
Nice! Way to press through
Thanks!

I'll work on deduplicating some parts of the code soonish (as discussed sometime during the review)
BUG: Unclear error message when merging tables and passing invalid option to validate
State: closed

- [ ] closes #49417
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Thanks @raisadz (failures are unrelated)
issue 48855 enable pylint unnecessary-pass
State: closed

Issue #48855. This PR enables pylint type "W" warning: `unnecessary-pass`.
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Hi. During CI occurs one Error
> intersphinx inventory 'https://docs.scipy.org/doc/scipy/objects.inv' not fetchable due to <class 'requests.exceptions.ConnectionError'>: HTTPSConnectionPool(host='docs.scipy.org', port=443): Max retries exceeded with url: /doc/scipy/objects.inv (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fd5800db8e0>: Failed to establish a new connection: [Errno 111] Connection refused'))

It doesn't look related to my changes.
API: make some Timestamp args keyword-only
State: closed

- [x] closes #45307 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I think this will make it easier to get #49365 working.

Labels:
name: API Design
name: Timestamp
description: pd.Timestamp and associated methods

Comments:
Could you see if this addresses https://github.com/pandas-dev/pandas/issues/32526, https://github.com/pandas-dev/pandas/issues/32174?
Definitely doesn't fix #32174.  looks like it makes #32526 better but that is a bit fuzzy on when we'd call that "solved"
Just a heads up, I think you linked `pull/32174` instead of https://github.com/pandas-dev/pandas/issues/32174
Thanks @jbrockmendel (makes `Timestamp.__new__` more sane)
DEPR: non-keyword args, errors arg
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel
DEPR: indexing
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel 
This is failing the "minimum versions" CI job (both in this PRs CI and in the one from `main`) https://github.com/pandas-dev/pandas/actions/runs/3373561158/jobs/5609678191
Fix Scorecards GitHub Action
State: closed

Fixes sudden failure reported in https://github.com/pandas-dev/pandas/issues/48566#issuecomment-1296174657.

Bumps ossf/scorecard-action to v2.0.6, fixing an unexpected breaking change in upstream API.


Labels:
name: CI
description: Continuous Integration

Comments:
this doesn't currently run on PRs - should it?

and does
```
# if: github.repository == 'pandas-dev/pandas'  # don't run on forks
```
need to be commented out?
A mistake. I've just pushed a new version undoing that comment and some accidental linting.

Also, the Action is currently still in development for PRs. I'll be sure to return here when it is made available.
Thanks @pnacht 
@pnacht this is still failing https://github.com/pandas-dev/pandas/actions/runs/3381132789/jobs/5614699256
shall we remove this until it's a bit more stable?
@MarcoGorelli maybe we can add `continue-on-error: true` for this job?
I think that's already the case - even if it fails, it's not blocking the other jobs from running
Hey everyone, thanks for letting me know. This one's a bit more surprising; I just created a [fork](https://github.com/pnacht/pandas/actions) of the project and the Action is running fine there... other projects with the Action also aren't having this issue.

I've also noticed it's [working again](https://github.com/pandas-dev/pandas/actions/workflows/scorecards.yml)... most odd.
BUG: dataframe construction with dict of recarrays should raise
State: closed

- [x] closes #49388
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I've added this to the whatsnew v1.5.2 as the first entry there. I don't know if there is actually a plan for pandas v1.5.2.


Labels:

Comments:
close temporarily
BUG/PERF: Series.replace with dtype="category"
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.0.0.rst` file if fixing a bug or adding a new feature.


Refactor of `Categorical._replace` to fix a few bugs with `Series(..., dtype="category").replace` and improve performance.


BUG 1: overlap between `to_replace` and `value`:

```
Series([1, 2, 3], dtype="category").replace({1:2, 2:3, 3:4})


# main:

0    4
1    4
2    4
dtype: category
Categories (1, int64): [4]


# PR:

0    2
1    3
2    4
dtype: category
Categories (3, int64): [2, 3, 4]
```

BUG 2: losing nullable dtypes of underlying categories:

```
Series(["a", "b"], dtype="string").astype("category").replace("b", "c")


# main:

0    a
1    c
dtype: category
Categories (2, object): ['a', 'c']


# PR:

0    a
1    c
dtype: category
Categories (2, string): [a, c]
```

Perf improvements:

```
import pandas as pd
import numpy as np

arr = np.repeat(np.arange(1000), 1000)
ser = pd.Series(arr, dtype="category")

%timeit ser.replace(np.arange(200), 5)

681 ms ± 9.65 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)   <- main
11 ms ± 690 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  <- PR
```

"""

Labels:
name: Bug
name: Refactor
description: Internal refactoring of code
name: Performance
description: Memory or execution speed performance
name: Categorical
description: Categorical Data Type

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
hopefully ill get time to take a good look this week but dont wait for me if this is a blocker.
Thanks @lukemanley 
This broke the ci, opened https://github.com/pandas-dev/pandas/pull/50848 to revert if there is no quick-win here
DEPR: core.index file
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel (looks like the failure was the unknown ASV ones)