BrandonFors/pandas-github-issues huggingface dataset:
TST/CI: Reduce number of unnecessary numba tests in groupby ops
State: closed

- [x] closes #42236
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Limiting the numba `engine_kwargs` parameterization for groupby tests similar to #42024

cc @mzeitlin11 

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: numba
description: numba-accelerated operations

Comments:
LGTM, that was fast haha
thanks @mroeschke and @mzeitlin11 for the report.
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
BUG: OutOfBoundsTimedelta not raised for out-of-bounds Timedelta 
State: closed

- [x] closes #36615 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Timedelta
description: Timedelta data type

Comments:
can you merge master
@jbrockmendel ok here
LGTM
thanks @Joeperdefloep
PERF/REF: groupby sample
State: closed

- [x] closes #34483
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Benchmarks:

```

       before           after         ratio
     [d61ace50]       [7fb839ce]
     <master~4>       <gb_sample>
-         116±5ms         29.8±3ms     0.26  groupby.Sample.time_sample
-        808±20ms         75.9±1ms     0.09  groupby.Sample.time_sample_weights

```


Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby
name: Performance
description: Memory or execution speed performance

Comments:
very nice @mzeitlin11
PERF: Avoid copy in DataFrame._reduce corner cases
State: closed

This implements BlockManager._get_data_subset (which ArrayManager already has) to avoid making a copy in a `DataFrame.__getitem__` usage inside DataFrame._reduce.

This path is deprecated so this snippet will be removed in 2.0.



Labels:
name: Performance
description: Memory or execution speed performance

Comments:
can you point to the benchmarks which change ? 
Not aimed at any benchmarks in particular, i just have a note to self to avoid a copy here.
PERF: tighten _should_compare for MultiIndex
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
is there a specific benchmark that this improves?
> is there a specific benchmark that this improves?

No.  The motivation is yak-shaving that traces back to getting rid of _convert_list_indexer
@jreback gentle ping (a whole mess of MultiIndex PRs yak-shaving inconsistencies)
@jbrockmendel  This PR seems to have introduced a bug. I verified this with the code below and a bisect.

```
import pandas as pd
mi = pd.MultiIndex.from_product([["a","b","c"],[1,2,3],["z","y","x"]])
df = pd.DataFrame(index=mi,dtype=float)
mi2 = pd.MultiIndex.from_product([["a","b","c"],[1,2,3]])
s = pd.Series(index=mi2, dtype=float)
s.iloc[:]=3.14
df["new"] = s
print(df)
```

now returns 
```
        new
a 1 z  NaN
    y  NaN
    x  NaN
  2 z  NaN
    y  NaN
    x  NaN
  3 z  NaN
    y  NaN
    x  NaN
```

Before this patch is did a broadcast assignment to the remaining MultiIndex levels, i.e..


```
        new
a 1 z  3.14
    y  3.14
    x  3.14
  2 z  3.14
    y  3.14
    x  3.14
  3 z  3.14
    y  3.14
    x  3.14
```
xref #40186
REF: implement Index._should_partial_index
State: closed

Formalize the IntervalIndex special case, in principle let subclasses override.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Interval
description: Interval data type
name: Index
description: Related to the Index class or subclasses

Comments:
can you rebase
Rebased + green
Is this issue still open?
> Is this issue still open?

No, this PR has been merged.
DOC: GH42204: Change docstring for head and tail in core/generic.py
State: closed

- [X] closes #42204
- [X] As test, ran scripts/validate_docstring.py with outputs
Docstring for "pandas.core.generic.NDFrame.head" correct. :)
Docstring for "pandas.core.generic.NDFrame.tail" correct. :)


Labels:
name: Docs
name: Stale

Comments:
Thanks for spotting the wrong usage of "function". For the examples I think it would be nicer to keep things as they are.
So what do you think about a different suggestion:

in head docstring:

For a negative integer `n = -k`, this function returns all rows except
the last `k` rows, equivalent to ``df[:-k]``.

in tail docstring:

For a negative integer `n = -k`, this function returns all rows except
the first `k` rows, equivalent to ``df[k:]``.

and no changes above the examples?

(If yes, would it be best to do this in a new pr?)
I find that new wording harder to understand than your original, but curious if others have any thoughts (regardless, you can always just push new commits to this pr instead of opening a new one)
+1 one on preferring the original wording (once modified based on existing feedback). Not sure it's necessary to adjust the examples - they use `n` (and that `n = -3` seems clear to me) instead of `k` so I don't find them confusing.
To be honest I find the original wording (current master) clearer, except for needing to insert `|n|` instead of `n` in one place. Albeit the meaning is obvious.

However, if these new changes are to be implemented I would suggest using the current implementation:

 - change "the function ``df.head(-k)``" to just "``df.head(-k)``"
 - change `k` back to `n`.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Thanks for the PR, but seems like the original wording with some modifications is preferable to what is here currently. Additionally, looks like this PR has gone stale. Going to close, but please ping if you're still interested in working on this and we can reopen.
DEPR: Series.__setitem__ with Float64Index falling back to positional
State: closed

- [x] closes #33469
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Deprecate
description: Functionality to remove in pandas

Comments:
@jorisvandenbossche ok here?
Looks good to me.

Can you add an explicit test (eg with the example series from the issue) that checks the warning is raised if the key (as float) is not present and does not raise a warning when it is present (that case that already works correctly label-based right now)?

There are no other index types that can end up in this case and trigger the warning?

> Can you add an explicit test (eg with the example series from the issue) that checks the warning is raised if the key (as float) is not present and does not raise a warning when it is present (that case that already works correctly label-based right now)?

Good idea, will do.

> There are no other index types that can end up in this case and trigger the warning?

Brainstorming I came up with 2 more cases; will update the whatsnew

```
msg = "Treating integers as positional in Series.__setitem__"

ii = IntervalIndex.from_breaks(range(10))[::2]
ser2 = Series(range(len(ii)), index=ii)
expected2 = ser2.copy()
expected2.iloc[-1] = 9

with tm.assert_produces_warning(FutureWarning, match=msg):
    ser2[4] = 9
tm.assert_series_equal(ser2, expected2)

mi = MultiIndex.from_product([[1.1, 2.1, 3.0, 4.1], ["A", "B"]])
ser3 = Series(range(len(mi)), index=mi)
expected3 = ser3.copy()
expected3.iloc[4] = 99

with tm.assert_produces_warning(FutureWarning, match=msg):
    ser3[4] = 99
tm.assert_series_equal(ser3, expected3)
```
i need to re-look can you rebase
BUG: excelwriter engine_kwargs fix
State: closed

- [x] closes #29095
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them - *I cannot find the linked page*
- [ ] whatsnew entry

The `engine_kwargs` argument was not passed properly and this was not tested. Also added some documentation.

Labels:
name: Docs
name: IO Excel
description: read_excel, to_excel
name: Stale

Comments:
Hello @Joeperdefloep! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-07-12 21:54:21 UTC
We would really love to have this fix in the next version of pandas, since we have to mitigate XLSX injections. Is there something I could help here to achieve this?
@Joeperdefloep can you merge master
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@Joeperdefloep do you still want to work on this? 
Thanks for the PR, but it appears to have gone stale. Let us know if you're still interested in working on this and we can reopen. Closing.
@mattelacchiato - any interest in picking this up?
Heyy, sorry I was on holidays a bit so that's why it stopped and went stale...

Also there was a weird thing that there was a merge conflict, where my test was changed to another one. That's why this PR doesn't mention it.

I will be at my computer tomorrow and hopefully will come back to this this week :)
DOC: add example to reorder levels
State: closed

- [ x] closes #42124
- [ ] tests added / passed
- [x ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ x] whatsnew entry - Included example for reorder_levels


Labels:
name: Docs

Comments:
Hello @prabha-git! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-06-25 23:10:03 UTC
closing for now as stale, thanks anyway for your PR!
TYP: update to mypy-0.910
State: closed


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
we will be able to start using TypeGuard https://docs.python.org/3.10/library/typing.html#typing.TypeGuard (imported from typing_extensions inside a TYPE_CHECKING)

Unfortunately, we can't use overloads or preserve types and most of our casts will still be required as the narrowed type will be wider in most cases!
nice!
Backport PR #42200 on branch 1.3.x (TYP: update for numpy-1.21.0)
State: closed

Backport PR #42200: TYP: update for numpy-1.21.0

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
i'll push some changes for the mypy failures shortly (or can merge and i'll open a PR against 1.3.x to fix)
TYP: update for numpy-1.21.0
State: closed


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
The MaskedRecords change is the only one that looks questionable, everything else I think we can merge and cleanup in follow-ups.
> The MaskedRecords change is the only one that looks questionable

sure. will look at that in detail tomorrow. feel free to push changes here in the meantime.
may need to do a 1.2.6 with this unfortunately 
@simonjayhawkins we need for 1.3 i assume.
just pushed restoring the MaskedRecords annotation
> @simonjayhawkins we need for 1.3 i assume.

not strictly, but best not to diverge the ci before we release

> may need to do a 1.2.6 with this unfortunately

There are no behavioral code changes that indicate that there are issues with 1.2.5 and numpy 1.21 
`test_read_csv` failing on Windows py38_np18 is not down to changes here

> > may need to do a 1.2.6 with this unfortunately
> 
> There are no behavioral code changes that indicate that there are issues with 1.2.5 and numpy 1.21

we had testing on 1.2.x for numpy-dev, but if we do get reports of issues with 1.2.5 and numpy 1.21 we may well want to do a 1.2.6 to fix (and then backport these changes to 1.2.x also, 1.2.x is on mypy 0.782 but probably not an issue)
PERF: IntervalArray.unique, IntervalIndex.intersection
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Partially address perf hit discussed in #41929 

```
import numpy as np
from pandas import *

N = 10**5

left = np.append(np.arange(N), np.array(0))
right = np.append(np.arange(1, N + 1), np.array(1))
intv = IntervalIndex.from_arrays(left, right)

intv2 = IntervalIndex.from_arrays(left + 1, right + 1)

%timeit intv.intersection(intv2)
122 ms ± 2.34 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <- PR
263 ms ± 5.98 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <- master
```

xref #36482

Labels:
name: Interval
description: Interval data type

Comments:
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
BUG: nullable integer, floating arrays raising from booleans
State: closed

- [x] closes #42137
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Closing pending further discussion on #42198
DOC: Correct docstring for Series.apply
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Ref: https://github.com/pandas-dev/pandas/pull/39941#issuecomment-829604511 and the conversation preceding it. While I don't find @MarcoGorelli's suggestion as implemented here very clear, it at least makes it correct and I don't see a better way.

Labels:
name: Docs

Comments:
@MarcoGorelli The docstring was incorrect before #39941 - I think we should value a correct docstring here even if it's not exactly clear what the behavior is. To be sure, while I find this unclear, my opinion is that it's the implementation (changing behavior based on whether an EA has certain private method or not) that is undesirable, but I don't see a clear way to fix.
thanks @rhshadrach 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
DOC: Move Other bugfixes to appropriate sections
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them



Labels:
name: Docs

Comments:
hmm mypy is failing oddly
Comparing to the most recent commit on master, I'm seeing numpy 1.21.0 here and 1.20.3 there.
CI failure is unrelated (ref #42196), @jreback and @simonjayhawkins for review.
thanks @rhshadrach 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
BUG: Styler.hide_columns() does not hide index name row
State: closed

- [ ] closes #42101 
- [ ] tests not added. Will update this PR with the tests.



Labels:
name: Styler
description: conditional formatting using DataFrame.style

Comments:
Hello @raghuabhishek! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/io/formats/style_render.py`](https://github.com/pandas-dev/pandas/blob/3d7985ffedd751ac7d9eb588dedcf1a15038fcb5/pandas/io/formats/style_render.py):

> [Line 353:1](https://github.com/pandas-dev/pandas/blob/3d7985ffedd751ac7d9eb588dedcf1a15038fcb5/pandas/io/formats/style_render.py#L353): [E101](https://duckduckgo.com/?q=pep8%20E101) indentation contains mixed spaces and tabs
> [Line 353:1](https://github.com/pandas-dev/pandas/blob/3d7985ffedd751ac7d9eb588dedcf1a15038fcb5/pandas/io/formats/style_render.py#L353): [W191](https://duckduckgo.com/?q=pep8%20W191) indentation contains tabs
> [Line 353:8](https://github.com/pandas-dev/pandas/blob/3d7985ffedd751ac7d9eb588dedcf1a15038fcb5/pandas/io/formats/style_render.py#L353): [E131](https://duckduckgo.com/?q=pep8%20E131) continuation line unaligned for hanging indent



##### Comment last updated at 2021-06-23 14:24:46 UTC
> Will update this PR with the tests.

It might be enough to correct the failing test actually.


This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
CLN: simplify Styler copy mechanics and tests
State: closed

checks an items off #40675 

The current mechanics are unstructured and the tests is not future proof (if new attributes are added tests will not pick these up and fail). This PR generalises the construction and future proofs the tests. No user facing cahnge.


Labels:
name: Refactor
description: Internal refactoring of code
name: Clean
name: Styler
description: conditional formatting using DataFrame.style

Comments:
thanks @attack68
STYLE: moving spacing conventions in cython casting to pre-commit
State: closed

- [X] closes #42121 
- [ ] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
> This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

Working
are you sure? looks like there's still lot's of failures https://github.com/pandas-dev/pandas/pull/42186/checks?check_run_id=3164816180
Sorry for the late reply, I think it's because the check is working on .rst files as well according to the include statement in the .yaml file:

https://github.com/pandas-dev/pandas/blob/2f49b8d32dc579358c63341730854f856ceb4315/.pre-commit-config.yaml#L96-L108

What should we do? separate this check with another id for only cython files?
Ah yes, you're right. Then yes, just make another hook and only run it on `.pyx` and `.pxi.in` files in `pandas/_libs` 
> sorry, forgot this last time, we should also remove 'lint' from this file (there's a few places where it crops up)
> 
> https://github.com/pandas-dev/pandas/blob/4f0e2e9712dd06d3e445c5777fd979c21cd84c02/ci/code_checks.sh#L13

Made the changes, should be good to go
oops, CI checks are failing, there's still 

https://github.com/pandas-dev/pandas/blob/4f0e2e9712dd06d3e445c5777fd979c21cd84c02/.github/workflows/ci.yml#L51-L53

(after that, should be good to go!)
Need to make changes in [docs](https://github.com/01-vyom/pandas/blob/pandas-42121/doc/source/development/contributing_codebase.rst#code-standards) as well.
There are also docs for pattern matching, but it has been moved so should I change it as well?

[Doc1](https://github.com/01-vyom/pandas/blob/pandas-42121/doc/source/development/contributing_codebase.rst#code-standards)

Doc2: 

https://github.com/pandas-dev/pandas/blob/4f0e2e9712dd06d3e445c5777fd979c21cd84c02/ci/code_checks.sh#L5-L9

Thanks for spotting that - yeah, sure!
TST: Add unmatched warning messages to assert_produces_warning() error
State: closed

- [x] closes #[42103](https://github.com/pandas-dev/pandas/issues/42103)
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

I didn't add anything to the whatsnew file, as it seems it's not always necessary depending on the change?

Error message for a failed assert_produces_warning(match=". . .") now looks like e.g.:
```
Did not see warning 'DtypeWarning' matching 'Warning'. The emitted warning messages are [DtypeWarning('This is not a match.'), DtypeWarning('Another unmatched warning.')]
```

Thanks for any feedback if I've done something wrong here.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Hello @calvinsomething! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-06-24 13:13:28 UTC
Nice! Looks like there's going to be a pre-commit failure though, did you run it locally? See https://pandas.pydata.org/docs/development/contributing.html#pre-commit

Can the test names be slightly more descriptive, e.g. test_fail_to_match_future_warning for the first and likewise for the others?

Other than that, looks good to me - thanks for doing this, it'll be useful to anyone testing warnings!
Thanks! No I didn't run pre-commit this time. I will make the changes and make the checks before adding the next commit.
> I didn't run pre-commit this time

if you do `pre-commit install` then it'll run automatically for you before each commit
Hopefully that looks better!
Sorry. Ran pre-commit manually before, but now I have it installed and have corrected the flake-8 error. Everything should be perfect... hopefully.
Made the last suggested change and now all the warning messages are more conformed.
![](https://i.fluffy.cc/MKjDMmwXxZlqRTwnn070gVqZ8Rwh6d3p.gif)
BUG: json_normalize not consistently ignoring errors (#41876)
State: closed

- [x] closes #41876
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
cc @WillAyd 
I'll be adding some unit tests as well
@jbrockmendel @WillAyd mind kicking off the additional workflows/taking another look? thanks!
also a conflict on the release notes
@jreback should be good now!
thanks @neelmraman
TST/CI: speed up chunksize test
State: closed

Seeing lots of CI timeouts - this test sometimes takes > 5 min for the python parser, guessing dependent on memory availability. 

AFAICT the data size is only so large to test the warning given by the c parser and low memory case, so shrank size for other cases.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO CSV
description: read_csv, to_csv
name: CI
description: Continuous Integration

Comments:
Also tentatively removed resource warning handling as hoping that was only needed because of memory issues.
thanks @mzeitlin11 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.