BrandonFors/pandas-github-issues huggingface dataset:
BUG: rolling with Int64
State: closed

- [x] closes #43016
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Window
description: rolling, ewma, expanding
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
thanks @mroeschke 
FWIW:

```
In [1]: import numpy as np

In [2]: import pandas as pd

In [3]: from pandas.core.dtypes.common import *

In [4]: arr = np.arange(5, dtype=np.int64)

In [5]: %timeit np.asarray(arr, dtype=np.int64)
150 ns ± 14 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

In [6]: %timeit arr.astype(np.int64, copy=False)
181 ns ± 13.3 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

In [7]: %timeit ensure_int64(arr)
63.1 ns ± 8.89 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
```

The absolute cost is very small, but its optimized to the bone.
I think this also fixed my issue in #43381. Any chance it can be backported to the 1.3.x branch?
We backport only regression fixes
TST: Add testing to df.where() typecasting as per GH 42295
State: closed

- [x] closes #42295
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

This makes sure that df.where does not cast floats to integers.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @maximilianaccardo! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-08-26 01:52:01 UTC
Thanks @maximilianaccardo. Happy to have you tackle more issues that require tests.
BUG: Pass index data correctly in groupby.transform/agg w/ engine=numba
State: closed

- [x] closes #43133
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Groupby
name: numba
description: numba-accelerated operations

Comments:
@mroeschke would be ok with backing this to 1.3.x if ok with that (move release note).
Sure moved to 1.3.3
thanks @mroeschke 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
PERF: Rolling._apply
State: closed

- [x] closes #42736
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

```
from asv_bench.benchmarks.rolling import *

self = Engine()
self.setup("DataFrame", "float", sum, "cython", "sum")
self.time_rolling_methods("DataFrame", "float", sum, "cython", "sum")

%timeit self.time_rolling_methods("DataFrame", "float", sum, "cython", "sum")
150 µs ± 889 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <- PR
224 µs ± 3.18 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <- master
```

cc @mroeschke 

Labels:
name: Performance
description: Memory or execution speed performance
name: Window
description: rolling, ewma, expanding

Comments:
There's an `_apply_series` which looks like a 1D case of `_apply_blockwise`; can that be refactored into `_apply_blockwise`?

`_apply_blockwise` maybe should be renamed to `_apply_columnwise`
Updated timings:

```
from asv_bench.benchmarks.rolling import *

self = Engine()
self.setup("DataFrame", "float", sum, "cython", "sum")
self.time_rolling_methods("DataFrame", "float", sum, "cython", "sum")

%timeit self.time_rolling_methods("DataFrame", "float", sum, "cython", "sum")
209 µs ± 8.64 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <- master
147 µs ± 3.32 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <- PR, consolidate=True
133 µs ± 1.95 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <- PR, consolidate=False
```

I'm OK with consolidating at that cost.
> Does this deserve a whatsnew note?

either way i guess, i can add something in the next "assorted follow-ups" PR
Sure that sounds good. Thanks @jbrockmendel
TST: DataFrame.mad with nullable dtype
State: closed

- [x] closes #33036
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @jbrockmendel
Pin aiobotocore to get ci passing
State: closed

- [x] xref #43168

Aiobotocore got an upgrade from 1.3.3 to 1.4.0, looks like this broke our ci

This comes with s3fs. The new release handles the dependency, but is not on conda-forge yet

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
Looks ok if green - is there an upstream issue we can link on what exactly is broken? 
Only found https://github.com/dask/s3fs/blob/main/requirements.txt
@meeseeksdev backport 1.3.x
Fixed all warning.warn messages for space,full-stop etc issues
State: closed

I have fixed all the warning.warn messages in the entire code base for "Minor typo on warnings #39158" reported issue.

closes  #39158

Summary of the changes .
[Fix-Tracking-Sheet (2).xlsx](https://github.com/pandas-dev/pandas/files/7030053/Fix-Tracking-Sheet.2.xlsx)


Labels:
name: Docs

Comments:
Hello @Kavya9986! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:















































































There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-08-25 09:55:36 UTC
Hi @Kavya9986 - few tests are failing because the old error messages are used in them. You might want to update the error msgs as well.
You can find more details of the failed tests [here](https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=64742&view=ms.vss-test-web.build-test-results-tab).
You may also run the tests locally as mentioned in the [doc](https://pandas.pydata.org/pandas-docs/stable/development/contributing_codebase.html#running-the-test-suite).
> Hi @Kavya9986 - few tests are failing because the old error messages are used in them. You might want to update the error msgs as well.
> You can find more details of the failed tests [here](https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=64742&view=ms.vss-test-web.build-test-results-tab).
> You may also run the tests locally as mentioned in the [doc](https://pandas.pydata.org/pandas-docs/stable/development/contributing_codebase.html#running-the-test-suite)

Hi !
Thanks. This is very useful . I will try fixing the error messages . 
Yes . I ran a script .
cool, do you want to share it? perhaps a pre-commit check could be made out of it
List of files changed.
[files_changed.txt](https://github.com/pandas-dev/pandas/files/7047463/files_changed.txt)

Script
```
#!/bin/bash

while IFS= read -r LINE; do
    echo $LINE
    pre-commit run flake8 --files "$LINE"
done < files_changed.txt

# files_changed.txt are the list of files that where changed since last commit

```


I meant, to find the warnings which needed changing - I don't think `pre-commit run flake8 --files` will tell you that?
Oh ! Initially I tried automating it fully , but did it partially as I thought return on my time investment would not be justified .

So first I ran this command to find the files and line number where potential fix was required 
`grep -rn 'warnings.warn(' . | cut -d":" -f1,2 | tr ":" "," > ../fix_tracking_sheet.csv`

And then filtered out all the .c files as all the warning messages there where commented .

After which I used this `grep -n8 'warnings.warn(' <File Name>` to do quick checks on each directory , fixed wherever required .

I'm still trying to use some kind of pattern-matching to automate the whole task :)

				

> I meant, to find the warnings which needed changing - I don't think `pre-commit run flake8 --files` will tell you that?
PERF: DataFrame.values
State: closed

```
from asv_bench.benchmarks.frame_methods import *
self = ToNumpy()
self.setup()

%timeit self.time_to_numpy_mixed_tall()
24.2 ms ± 324 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <- master
11.8 ms ± 609 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <- PR
```

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
cool maybe have a perf improvement note listing a bunch of issues would be good
TST: Test inequality w/ complex dtypes (GH28050)
State: closed

- [x] closes #28050
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Whats new::
Add a test to `pandas/test/dtypes/test_dtypes.py`


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@phofl thanks for the comments please see revision
Thanks @calvh. The CI failures were unrelated.
ENH: use github issue form for doc improvement
State: closed

xref #42951, #43009

Screenshot of current version: 
<img width="829" alt="Screen Shot 2021-08-21 at 12 12 44 PM" src="https://user-images.githubusercontent.com/37011898/130328264-29ea829e-b664-4a68-bb7d-8a3b21c5bfee.png">


https://github.com/mzeitlin11/issue-forms/issues if anyone wants to play around with opening issues with the form.

Labels:
name: Docs
name: Admin
description: Administrative tasks related to the pandas project

Comments:
Thanks @mzeitlin11
CI: use pipx and build to simplify and remove a workaround
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


This cleans up the SDist build slightly. First, it uses PyPA's "build" instead of directly calling `setup.py`, which will respect PEP 518 requirements and avoids the workaround of installing NumPy to be able to run setup.py. It also uses pipx to avoid needless setup; PyPA's pipx is provides as a first-class package manager on GitHub Actions (and Azure, they share images).


Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration
name: Stale

Comments:
> PyPA's pipx is provides as a first-class package manager on GitHub Actions (and Azure, they share images).

this doesn't help as the archives are not generated here (they are actually justed tested here). sure i suppose we could move to this workflow, but that should happen in conjuction with a change like this.
Where are the SDist's built? https://github.com/MacPython/pandas-wheels only does wheels, AFAICT.

Also, this is still testing that the artifact builds and is installable, it's just using better infrastructure for doing so without manual workarounds in producing them.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@simonjayhawkins do we have any manual process in the release for sdists? i don't recall if this changed recently.
note completely averse here but this would need to address comments & rebase
Thanks for the PR, but it appears to have gone stale. If interested in continuing please merge the main branch, address the review, and we can reopen.
I've rebased, but you can't reopen a PR after force pushing to it (IIRC).  I think this was stuck because there was some "manual/other release process" that also needed updating but I wasn't pointed to that.
Ah okay. Yes I see we can't reopen this PR (we also don't recommend force pushing to branch as it makes the commit history and changes harder to view).

If you'd like to continue with a new PR we can try to contact the right person who has set up sdist.
There weren't really any changes yet, it was just tracking the current main. I can retry.
> @simonjayhawkins do we have any manual process in the release for sdists? i don't recall if this changed recently.

yes, the sdist is created locally during the release process and then uploaded to the github release artifacts so that conda-forge can use it and also uploaded to PyPI

> Where are the SDist's built? https://github.com/MacPython/pandas-wheels only does wheels, AFAICT.

https://github.com/pandas-dev/pandas-release

and the conda recipe that points to the manually created sdist uploaded as a github release artifact is at 

https://github.com/conda-forge/pandas-feedstock
TST: groupby.first/last retains categorical dtype
State: closed

- [x] closes #33090
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Categorical
description: Categorical Data Type

Comments:
thanks @debnathshoham 

since we get a lot of PRs, periodically rebasing yours is a good idea as it moves them to the front of the queue (assuming they are ready to go)
BUG: Outer/right merge with EA dtypes cast to object
State: closed

- [x] closes #40073
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=64682&view=logs&j=bf526160-cc38-5046-7b49-5141c64a483a&t=6a92c4fe-2b0b-5acd-cfba-e7901042d9bd is failing. maybe restrict the test to only EA types e.g. any_float_dtype and any_int_ea_dtype
Hi @jreback - The failure is somehow in TestMerge.test_handle_join_key_pass_array. And not the ones I added. Surprisingly, its not failing in my local
> Hi @jreback - The failure is somehow in TestMerge.test_handle_join_key_pass_array. And not the ones I added. Surprisingly, its not failing in my local

Since these are windows and 32-bit failures, would guess it has to do with these platforms defaulting to i32 instead of i64
> Since these are windows and 32-bit failures, would guess it has to do with these platforms defaulting to i32 instead of i64

If these are defaulting to int32 instead of int64, shouldn't the assertion pass? (the way both defaults to int64 in my local).
@mzeitlin11 if you can give a quick relook
thanks @debnathshoham 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
REF: `axis in [..]` with `_get_axis_number()` 
State: closed

- [x] xref #43048 
- [x] Refactored test_frame_apply.py 
- [x] Created if-else constructs by replacing `if axis in [...]` by `_get_axis_number`

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Done. Please check.
Thank you.
lgtm pending green
BUG: GroupBy.quantile fails with pd.NA
State: closed

- [x] closes #42849
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
@jreback @jbrockmendel Green
Hi @jreback , @jbrockmendel - could you please take a look if this looks fine
Hi @jbrockmendel could you pls take a quick look if this is fine now?
LGTM

wouldnt surprise me if some of the tests could be condensed/parametrized, OK for follow-up
thanks @debnathshoham (followup to parameterize tests here would be great, that can be on master)
TYP: callable refactor for apply methods in styler
State: closed

unrelated question: if i'm in pandas-core can I be added as a maintainer to `config.yml`?

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> unrelated question: if i'm in pandas-core can I be added as a maintainer to config.yml?

Yes, absolutely - this _should_ be part of the process of adding maintainers, but is often forgotten. Sorry for not having picked up on this sooner
> So, this is necessary because the return type isn't guaranteed to be `Styler`, right? If so, looks good to me

correct
PERF: _form_blocks
State: closed

```
from asv_bench.benchmarks.frame_ctor import *

self = FromArrays()

self.setup()

%timeit self.time_frame_from_arrays_int()
2.28 ms ± 67 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <- master
1.82 ms ± 5.34 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <- PR
```

Orthogonal to #43142, one more optimization after these two are in.

Labels:
name: Performance
description: Memory or execution speed performance
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
https://github.com/pandas-dev/pandas/pull/43144/checks?check_run_id=3386832245 looks like a real error
Huh can't reproduce the failures locally, will have to dig into this
OK, looks like in numpy<1.21 comparing np.dtype object to EA dtype object raises instead of returning False, so the itertools.groupby is failing.  will figure out a workaround
updated with np<1.21-compatiblity + green
@jreback gentle ping; hoping to get this in before you take off
REF: remove create_block_manager_from_arrays
State: closed

Looks like dask uses create_block_manager_from_blocks, but I haven't seen any evidence that any downstream package uses create_block_manager_from_arrays.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
lgtm. needs a rebase? (as not sure if this overlaps with other PRs)
shouldnt, no
DOC added example to reorder levels
State: closed

Added an example for reorder levels.

Labels:
name: Docs
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Hello @Nikhil636! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/core/frame.py`](https://github.com/pandas-dev/pandas/blob/9776e2e7fe6ce2b9b8d37eea63dc19abefe03268/pandas/core/frame.py):

> [Line 6840:1](https://github.com/pandas-dev/pandas/blob/9776e2e7fe6ce2b9b8d37eea63dc19abefe03268/pandas/core/frame.py#L6840): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 6846:42](https://github.com/pandas-dev/pandas/blob/9776e2e7fe6ce2b9b8d37eea63dc19abefe03268/pandas/core/frame.py#L6846): [W291](https://duckduckgo.com/?q=pep8%20W291) trailing whitespace
> [Line 6851:1](https://github.com/pandas-dev/pandas/blob/9776e2e7fe6ce2b9b8d37eea63dc19abefe03268/pandas/core/frame.py#L6851): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 6851:9](https://github.com/pandas-dev/pandas/blob/9776e2e7fe6ce2b9b8d37eea63dc19abefe03268/pandas/core/frame.py#L6851): [E101](https://duckduckgo.com/?q=pep8%20E101) indentation contains mixed spaces and tabs
> [Line 6851:9](https://github.com/pandas-dev/pandas/blob/9776e2e7fe6ce2b9b8d37eea63dc19abefe03268/pandas/core/frame.py#L6851): [W191](https://duckduckgo.com/?q=pep8%20W191) indentation contains tabs
> [Line 6853:26](https://github.com/pandas-dev/pandas/blob/9776e2e7fe6ce2b9b8d37eea63dc19abefe03268/pandas/core/frame.py#L6853): [W291](https://duckduckgo.com/?q=pep8%20W291) trailing whitespace
> [Line 6857:1](https://github.com/pandas-dev/pandas/blob/9776e2e7fe6ce2b9b8d37eea63dc19abefe03268/pandas/core/frame.py#L6857): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 6859:1](https://github.com/pandas-dev/pandas/blob/9776e2e7fe6ce2b9b8d37eea63dc19abefe03268/pandas/core/frame.py#L6859): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 6861:26](https://github.com/pandas-dev/pandas/blob/9776e2e7fe6ce2b9b8d37eea63dc19abefe03268/pandas/core/frame.py#L6861): [W291](https://duckduckgo.com/?q=pep8%20W291) trailing whitespace
> [Line 6865:1](https://github.com/pandas-dev/pandas/blob/9776e2e7fe6ce2b9b8d37eea63dc19abefe03268/pandas/core/frame.py#L6865): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace



##### Comment last updated at 2021-08-21 07:19:08 UTC
Actually, #42935 was already open for this issue, and it looks like it's still active - let's close in favour of that one for now

Next time, please check that there's not already an open PR, that one was linked in the issue

EDIT
----

This PR was a blatant attempt at plagiarising #42935
REF: Replacing `axis in` with `_get_axis_number(axis)`
State: closed

xref [#43048](https://github.com/pandas-dev/pandas/issues/43048). Refactored test_label_or_level_utils.py

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
Hello @RohanTrix! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-08-20 17:24:56 UTC
lgtm pending green tests
thanks @RohanTrix very nice. Leavning the issue open if there are additional cases.
Thanks @jreback !
REF: use _get_axis_number instead of if axis in [...]: #43048
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:

Comments:
Hello @ppulkit5! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/io/formats/style.py`](https://github.com/pandas-dev/pandas/blob/74d6019b4a676dcb143b39db573270ccec343632/pandas/io/formats/style.py):

> [Line 1153:27](https://github.com/pandas-dev/pandas/blob/74d6019b4a676dcb143b39db573270ccec343632/pandas/io/formats/style.py#L1153): [E271](https://duckduckgo.com/?q=pep8%20E271) multiple spaces after keyword


this is being addressed in #43121
REF: use _get_axis_number instead of if axis in [...]: #43048
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:

Comments:
this is being addressed in #43127.

Please find one of the many examples of code to fix that someone else has not already submitted a PR for.