BrandonFors/pandas-github-issues huggingface dataset:
ENH: mask support for hastable functions for indexing
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This is a first step towards creating a MaskedEngine to optimise performance in these cases. This supports masks in the HashTable.
Did not impact performance of non masked cases.

cc @jorisvandenbossche 

As a precursor to a MaskedEngine we have to adjust the HashTable methods to support masked na values. They keep track of seen na values separately. I think this makes more sense than keeping track of the NAs on the Engine level

``map_locations`` and ``lookup`` are normally called when the values are unique or used to check if the values are unique. Nevertheless, if they have duplicate values, they always return the index of the last occurrence of a specific value. This is the same in ``get_item`` and ``set_item``. We have to adjust the length of the HashTable, because the length is used to check if the values were unique. 


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Yes this is ready. Would add the actual Engine as a follow up
Thanks @phofl (can add follow up PRs if needed)
ENH: raise_assert_detail shows the difference between the columns
State: closed

- [x] xref https://github.com/pandas-dev/pandas/pull/48033 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This idea came out of https://github.com/pandas-dev/pandas/pull/48033 , so I'm opening a PR to see if anyone else would like to see this

---

Adding an extra `At index {idx}, diff: {left} != {right}`, analougously to pytest's output

Demo:
```python
import pandas as pd
import numpy as np

a = np.random.randint(0, 10, size=200)
b = np.random.randint(0, 10, size=200)
df1 = pd.DataFrame({'a': a, 'b': b})
df2 = pd.DataFrame({'a': a, 'b': b})
df2.loc[97, 'b'] = 42
pd.testing.assert_frame_equal(df1, df2)
```
outputs
```
AssertionError: DataFrame.iloc[:, 1] (column name="b") are different

DataFrame.iloc[:, 1] (column name="b") values are different (0.5 %)
[index]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, ...]
[left]:  [1, 4, 3, 6, 4, 7, 2, 5, 3, 6, 6, 9, 8, 4, 4, 4, 7, 8, 4, 0, 3, 1, 4, 5, 7, 0, 3, 0, 9, 9, 6, 1, 7, 7, 4, 7, 6, 2, 3, 9, 1, 5, 1, 7, 2, 3, 8, 9, 5, 3, 6, 3, 7, 3, 4, 1, 7, 3, 6, 9, 0, 0, 7, 3, 6, 4, 4, 1, 0, 8, 0, 1, 8, 2, 8, 1, 8, 8, 8, 3, 8, 3, 8, 5, 7, 0, 0, 2, 8, 5, 3, 4, 7, 1, 6, 3, 5, 9, 7, 8, ...]
[right]: [1, 4, 3, 6, 4, 7, 2, 5, 3, 6, 6, 9, 8, 4, 4, 4, 7, 8, 4, 0, 3, 1, 4, 5, 7, 0, 3, 0, 9, 9, 6, 1, 7, 7, 4, 7, 6, 2, 3, 9, 1, 5, 1, 7, 2, 3, 8, 9, 5, 3, 6, 3, 7, 3, 4, 1, 7, 3, 6, 9, 0, 0, 7, 3, 6, 4, 4, 1, 0, 8, 0, 1, 8, 2, 8, 1, 8, 8, 8, 3, 8, 3, 8, 5, 7, 0, 0, 2, 8, 5, 3, 4, 7, 1, 6, 3, 5, 42, 7, 8, ...]
At positional index 97 diff: 9 != 42
```

This is analogous to how `pytest` shows diffs between lists:
```
    def test_me():
        a = [1, 2, 3]
        b = [4, 5, 6]
>       assert a == b
E       assert [1, 2, 3] == [4, 5, 6]
E         At index 0 diff: 1 != 4
E         Use -v to get more diff
```

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @MarcoGorelli
TST: add tests for parser functions being pickle-able.
State: closed

- [x] closes #22265
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
Thanks @DriesSchaumont
ENH: Suppport masks in median groupby algo
State: closed

- [x] xref #37493 (Replace xxxx with the Github issue number)
- [] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Only a small speed up (5-10%, depending on the number of NAs). In general, this is needed to streamline the code base in ops.py after everything supports masks.


Labels:
name: Groupby
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @phofl
CLN/DOC: Adjust xpath validation and error messaging in read_xml with IO tools doc note and example
State: closed

- [X] closes #48181
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: Clean
name: IO XML
description: read_xml, to_xml

Comments:
Thanks @ParfaitG
PERF: avoid unnecessary reordering in MultiIndex._reorder_indexer
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v1.6.0.rst` file if fixing a bug or adding a new feature.


`MultiIndex._reorder_indexer` contains logic to determine if indexer reordering is required. This PR expands the logic a bit to identify additional cases where reordering is not necessary.

ASVs: 

```
       before           after         ratio
     [4b645ae5]       [07679a60]
     <main>           <multiindex-need-sort>
-        5.07±1ms          726±9μs     0.14  indexing.MultiIndexing.time_loc_all_bool_indexers(True)
-        8.13±2ms         709±20μs     0.09  indexing.MultiIndexing.time_loc_all_bool_indexers(False)
```

Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
Thanks @lukemanley
TYP: contextmanager expects a Generator
State: closed

`contextmanager` should expect a `Generator`, not an `Iterator`. `Iterator[...]` and `Generator[..., None, None]` are not the same, see https://github.com/python/typeshed/issues/2772

This might help @asottile a bit with https://github.com/python/typeshed/pull/7430. Basically, staled because it would impact too many projects - after this PR there is one fewer :)

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @twoertwein (would be cool if autotyping could have a rule for this :) )
CI: Pin mambaforge image
State: closed

- [x] xref #48382 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Just a temporary fix, but can not reproduce the failure on arm so hard to inverstigate


Labels:
name: CI
description: Continuous Integration

Comments:
Thanks @phofl
DOC: Clarify that objects dtype takes precedence in where
State: closed

- [x] closes #48373 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @mroeschke 
Should we add a whatsnew too?


Labels:
name: Docs

Comments:
>  Should we add a whatsnew too?

Sure, if so we can backport this PR.
@mroeschke  I think this is ready
Thanks @phofl
PERF: Add var to masked arrays
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
This patch may have induced a potential regression. Please check the links below. If any ASVs are parameterized, the combinations of parameters that a regression has been detected appear as subbullets. This is a partially automated message.

 - https://asv-runner.github.io/asv-collection/pandas/#series_methods.NanOps.time_func
   - func='var'; N=1000000; dtype='boolean'
Here is a more direct link (still need to workout how to incorporate parameterizations in the link):

https://asv-runner.github.io/asv-collection/pandas/#series_methods.NanOps.time_func?p-func='var'&p-dtype='boolean'&p-N=1000000
Hm guess boolean is special casy here. We could cast the array to float, this would get most of the regression back, but not sure if this is worth it
Taking the variance of Booleans seems pretty rare to me, but at the same time, the solution sounds very easy. I'm thinking to at least open a tracking issue.
BUG: masked mean unnecessarily overflowing
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Do we want to add mean to the Masked Array interface (I would say yes, but want to check anyway)? This would simplify things a bit.

cc @jorisvandenbossche 

Labels:
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Ah, nice, this is possible now because we require numpy >= 1.20 (the `where` keyword was only added later to `mean`)

> Do we want to add mean to the Masked Array interface (I would say yes, but want to check anyway)? This would simplify things a bit.

Yeah, I am not really sure why historically sum, prod, etc are methods, while mean is only handled inside `_reduce`
Yes, numpy now supports this in every version we need.

Ok great, confused me a little bit too
Added as method
DOC: how to_datetime %S differs from strptime
State: closed

- [ ] closes #47134
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Hello @heroesIV! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-09-03 13:25:02 UTC
This is my first PR. Do let me know if any changes are needed. Open to all feedback.
Thanks @heroesIV !
DOC: Reword DataFrame.assign
State: closed

I think wording is now a bit more precise, and the explicit
comparison with assignment helps to understand what this does.
I've also moved the note on multiple assignments into the **kargs
description because I think it really belongs there and is not
only a remark we should make further down in the docstring.

- [N/A] closes #xxxx (Replace xxxx with the Github issue number)
- [N/A] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [N/A] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [N/A] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: Stale

Comments:
I think the precision in the docs is fine, though if another maintainer could also take a look too.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
STYLE: upgrade codespell
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Hello @prakhar-198! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











* In the file [`pandas/tests/config/test_config.py`](https://github.com/pandas-dev/pandas/blob/435ed56afc25306175fbe9e542a5bef0aecbf3aa/pandas/tests/config/test_config.py):

> [Line 19:89](https://github.com/pandas-dev/pandas/blob/435ed56afc25306175fbe9e542a5bef0aecbf3aa/pandas/tests/config/test_config.py#L19): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (90 > 88 characters)


thanks for your PR, but this is a duplicate of #48364
Upgrade codespell version and fix misspelled words
State: closed

- [x] closes #48349

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Hey @MarcoGorelli, thanks so much for guiding me through this issue. I did a rebase to update this PR with latest commits on pandas, not sure if it was the right way. Do i have to do anything else so it gets merged?
that's all thanks, I'll just leave it open a bit in case others have comments
Thanks @lucasdamo. Very nice
Added timeout on request.get() for ensuring that if a recipient serve…
State: closed

We have detected this issue in the `main` branch of `pandas` project on the version with commit hash `f0814b`. This is an instance of an api usage issue.

**Fixes for api usage issues:**
In file: `web/pandas_web.py`, in method `roadmap_pdeps`, `home_add_releases` and `maintainers_add_info` a request is made without a timeout parameter. If the recipient server is unavailable to service the request, [application making the request may stall indefinitely. ](https://docs.python-requests.org/en/master/user/advanced/#timeouts)our intelligent code repair system **iCR** suggested that a timeout value should be specified.

This issue was detected by our **OpenRefactory's Intelligent Code Repair (iCR)**. We are running **iCR** on libraries in the `PyPI` repository to identify issues and fix them. You will get more info at: [pypi.openrefactory.com](https://pypi.openrefactory.com/)


Labels:
name: Web
description: pandas website
name: Stale

Comments:
Hello @openrefactory! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`web/pandas_web.py`](https://github.com/pandas-dev/pandas/blob/2b4d7cae6aacff23f5b9e2d6197c0f14d3b25b00/web/pandas_web.py):

> [Line 160:89](https://github.com/pandas-dev/pandas/blob/2b4d7cae6aacff23f5b9e2d6197c0f14d3b25b00/web/pandas_web.py#L160): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (149 > 88 characters)
> [Line 173:89](https://github.com/pandas-dev/pandas/blob/2b4d7cae6aacff23f5b9e2d6197c0f14d3b25b00/web/pandas_web.py#L173): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (141 > 88 characters)
> [Line 174:89](https://github.com/pandas-dev/pandas/blob/2b4d7cae6aacff23f5b9e2d6197c0f14d3b25b00/web/pandas_web.py#L174): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (100 > 88 characters)
> [Line 239:89](https://github.com/pandas-dev/pandas/blob/2b4d7cae6aacff23f5b9e2d6197c0f14d3b25b00/web/pandas_web.py#L239): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (141 > 88 characters)


Thanks. Adding the timeout is a reasonable change. Looks like there's some formatting issues
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Add missing pytest-asyncio to test extras
State: closed

While testing for #48360 locally I got some errors due to `pytest-asyncio` missing after I did:

```
pip install -e .[test]
```

This should fix it!


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Dependencies
description: Required and optional dependencies

Comments:
I'm not sure how to begin debugging the test failure :) Any tips?
@jorisvandenbossche, hoi! Sorry for the direct ping, I'm not sure who I should bother about this specific issue, and I'm not familiar with the typical PR cycle in pandas :) I'd like to not let this slip through the cracks: do you have pointers on how to solve the failures? Same goes for #48360.
Thanks for the pull request. It looks like this was incorporated in https://github.com/pandas-dev/pandas/pull/47336, so going to close in favor of that pull request. But thanks for the contribution!
FIX: Reject any non-array, non-ns datetimes and timedeltas
State: closed

Done at euroSciPy2022! Fixes #48312.

On suggestion of @jorisvandenbossche, we opted to raise an error instead of converting to `[ns]` in order to make it easier in the future to add support for other granularity levels.

co-author: @kislovskiy

- [x] closes #48312
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Stale
name: Timestamp
description: pd.Timestamp and associated methods

Comments:
Hello @brisvag! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-09-05 09:54:15 UTC
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
I don't know how to solve the build error without help, unfortunately, so I would appreciate some suggestions.
Thanks for the pull request, but it appears to have gone stale. Additionally, it appears this behavior may head in a different direction (https://github.com/pandas-dev/pandas/issues/48312#issuecomment-1278345347 - happy to have your input!) Closing until the discussion is resolved there