BrandonFors/pandas-github-issues huggingface dataset:
REF: eliminate inner functions in describe
State: closed

- [ ] xref #36833
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Remove inner functions in ``describe_ndframe``, while passing the required arguments.

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
thanks @ivanovmg
BUG: Series.__setitem__ with mismatched IntervalDtype
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Interval
description: Interval data type

Comments:
also pls add a whatsnew
rebased + greenish
ENH: Allow numpy ops and DataFrame properties as str arguments to DataFrame.apply
State: closed

- [x] closes #39116
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry



Labels:
name: Enhancement
name: Apply
description: Apply, Aggregate, Transform, Map
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
thanks @rhshadrach
DOC: NDFrame fillna method add use case
State: closed

- [x] closes #38917 
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

cc @MarcoGorelli

Labels:
name: Docs
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
thanks @BeanNan
TYP: replace Label alias from pandas._typing with Hashable
State: closed

- [ ] closes #37134
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
LGTM
TYP: fix mypy errors in pandas/core/arraylike.py
State: closed

- [ ] xref #37715
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Handle ignored mypy errors in ``pandas/core/arraylike.py``.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@ivanovmg if you can merge master and update
Thanks @ivanovmg I think the idea is to remove the ignores as they can be seen as a sort of TODO. adding the comment imo does not move this forward.

I think if you revert 8b8348621799ef64f1da2434c0e0f1f6f6874064. we can then just merge as was. see https://github.com/pandas-dev/pandas/pull/39257#discussion_r574442092
looks like @simonjayhawkins 's concerns have been addressed, so LGTM
thanks @ivanovmg
MOVE: describe to pandas/core/describe.py
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Move ``describe`` functionality to a separate module and prepare for the further refactor/simplification (xref #36833).

 - Created new module ``pandas/core/describe.py``
 - Delegate the functionality of ``NDFrame.describe`` to function ``describe_ndframe`` in the new module

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
@ivanovmg actually can you merge master again, think i just merged an incompatible change here
great ping on green
> great ping on green

@jreback, it is green. 
thanks @ivanovmg
TST: add test for describe include/exclude
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Added test for checking that if include is 'all' then setting exclude will raise ValueError.
This case was not covered.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thanks @ivanovmg
BUG: reindex raising TypeError with timezone aware index and tolerance for ffill and bfill
State: closed

- [x] closes #38566
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Is there a better place for these tests?

EDIT: Technically a regression, but since it persists since 1.0.0 is it worth backporting?

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Timezones
description: Timezone data dtype

Comments:
thanks @phofl yeah we shouldn't be backporting unless these are in 1.1
REF: implement dtypes.cast.can_hold_element
State: closed

Passes the tests added by #39086, gets us close to collapsing the NumericBlock subclasses and just using NumericBlock directly.

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
nice, let's merge that one first and rebase this.
ok give this a rebase
Rebased + green
TYP: NDFrame.pipe, GroupBy.pipe etc.
State: closed

Type up `pipe` function and methods. This allows type checkers to understand the return value type from pipes, which is nice.


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
cc @simonjayhawkins 
Updated.
I think this is ok to merge?
I think so. We may want to have an alias for this, but as the type is not yet complete I think we should leave that to later.

I say incomplete since we will replace the `...` of `Callable[..., T]` at some point, and that will require a second typevar to bind to self/obj.

but in the spirit of gradual typing, this gets the return type sorted.
.pipe takes very generic callables, so I don’t think it’s possible to type more. Can you give an example what you’re thinking?

In python 3.10 there’s [something coming](https://www.python.org/dev/peps/pep-0612/ ) that may improve typing for .pipe, but that’s far in the future for Pandas...
if func is a tuple, it is indeed not possible to bind to a keyword argument, but if func is not a tuple and args is not specified, then the function should take just 1 positional argument which is the object?

so one of the overloads would be

```
def pipe(
    obj: S, func: Union[Callable[[S], T], Tuple[Callable[..., T], str]], **kwargs
) -> T:
```
where S is a TypeVar and args not specified (args may need to be None, I can't remember off the top of my head)

Indeed, this is out of scope for now and I have approved the changes as is.
Ok. I think it’s ok to wait with that...
yeah, since we maybe pulling in some type stubs, https://github.com/pandas-dev/pandas/issues/28142#issuecomment-760549977, I suspect that we can't yet inline those stubs where the types are not compatible with Python 3.7 (without typing_extensions) and I suspect the overloads will be in stub file instead of inline ( and may be able to adopt PEP 612 sooner.)
Thanks @topper-123
CLN: Numpy compat functions namespace
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


Should we do something about the private variables ``_np_version_under1p19`` and ``_np_version_under1p20`` too?

cc @jreback

Labels:
name: Clean

Comments:
let's save this until we revert the numpy fixes
Sounds good, will fix the errors in the meantime. Will reopen when reverted 
Reopening since the other commit was reverted. Should we make ``_np_version_under1p19`` and ``_np_version_under1p20`` make non private to move them too?
> Reopening since the other commit was reverted. Should we make `_np_version_under1p19` and `_np_version_under1p20` make non private to move them too?

yes
Done, opened #39283
thanks @phofl
DOC: Added reverse rolling window examples
State: closed

- [x] closes #38627
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs
name: Window
description: rolling, ewma, expanding
name: Stale

Comments:
can you merge master and update
@vangorade can you merge master and update
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Going to close this PR due to lack of activity. Let us know if you'd like to continue working on it.
CI: Skip numpy dev failing tests
State: closed

- [x] xref #39089
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them



Labels:
name: CI
description: Continuous Integration

Comments:
After fixing the underlying cause we can simply revert this.

cc @jreback @simonjayhawkins 
right do the namespace changes first as a separate PR
ok let's just merge this until it's fixed
BUG: setting td64 value into Series[numeric]
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Timedelta
description: Timedelta data type

Comments:
lgtm i guess add a whatsnew note
CLN: Simplify optional dependency
State: closed

- [x] closes #38936 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
thanks @lithomas1 very nice
Remove xlrd benchmark after xlrd was updated to 2.0.1 on conda-forge
State: closed

- [x] closes #39076
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Is pinning here the right thing to do in environment.yml?

cc @jreback 

Labels:
name: CI
description: Continuous Integration
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
Alternatively we could remove this benchmark
Done
Have to backport this too.
great, backporting as well to avoid 1.2.x CI failures
@jreback greenish based on the changes here. Azure will take a while because of the new master commits
thanks @phofl 
@meeseeksdev backport 1.2.x
Backport PR #39065 on branch 1.2.x (Regression in replace raising ValueError for bytes object)
State: closed

Backport PR #39065: Regression in replace raising ValueError for bytes object

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Failure is caused by xlrd version fixed in #39077 and #39078 
> Failure is caused by xlrd version fixed in #39077 and #39078

yes these are totally fine. just waiting for the backport to pass first (the fix)
Fixed #39073 Add STUMPY to Pandas ecosystem docs
State: closed

- [ ] closes #39073
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
@jreback As we had briefly discussed, this is a PR to add STUMPY to the `ecosystem.rst` file. Thank you for the suggestion!
pre commit checks are complaining

```
@@ -88,9 +88,9 @@ Compose is a machine learning tool for labeling data and prediction engineering.
 `STUMPY <https://github.com/TDAmeritrade/stumpy>`__
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-STUMPY is a powerful and scalable Python library for modern time seris analysis. 
+STUMPY is a powerful and scalable Python library for modern time seris analysis.
 At its core, STUMPY efficiently computes something called a
-`matrix profile <https://stumpy.readthedocs.io/en/latest/Tutorial_The_Matrix_Profile.html>`__, 
+`matrix profile <https://stumpy.readthedocs.io/en/latest/Tutorial_The_Matrix_Profile.html>`__,
 which can be used for a wide variety of time series data mining tasks.
 
 .. _ecosystem.visualization:
```

ping when green.
@jreback The pre commit is green now
thanks @seanlaw
TST GH26807 Break up pandas/tests/io/pytables/test_store.py
State: closed

This PR addresses xref #26807 for pandas/tests/io/pytables/test_store.py, which was more than 5000 lines before this attempt. I made a new directory pandas/tests/io/pytables/store/ to put all the new modules in. I tried to 1) get all the new modules below 1000 lines and 2) break it up in a somewhat logical way, but let's see if I succeeded in the opinion of the reviewer.

There were 194 tests in pandas/tests/io/pytables/test_store.py, and now there are 194 tests in pandas/tests/io/pytables/store so I haven't gained or lost any. One is skipped on my dev machine.

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO HDF5
description: read_hdf, HDFStore
name: Code Style
description: Code style, linting, code_checks

Comments:
Hello @moink! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



















* In the file [`pandas/tests/io/pytables/test_select.py`](https://github.com/pandas-dev/pandas/blob/77c3af746b5bcecaf9b3a26ea1eac687d23602ec/pandas/tests/io/pytables/test_select.py):

> [Line 688:9](https://github.com/pandas-dev/pandas/blob/77c3af746b5bcecaf9b3a26ea1eac687d23602ec/pandas/tests/io/pytables/test_select.py#L688): [E741](https://duckduckgo.com/?q=pep8%20E741) ambiguous variable name 'l'







##### Comment last updated at 2021-01-13 17:00:26 UTC
this is a straight move right? e.g. no actual changes except to make tests / linting pass?
> this is a straight move right? e.g. no actual changes except to make tests / linting pass?

Correct. I made no changes at all to the tests - just moved existing ones to other classes & modules.
looks good. can you merge master (small conflict), and show the before/after test count. ping when ready
273 tests in pandas/tests/io/pytables both before and after the changes
thanks @moink
Regression in loc.setitem raising ValueError with unordered MultiIndex columns and scalar indexer
State: closed

- [x] closes #38601
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

@jbrockmendel Would it be faster to use arange too in the line above of my fix instead of Index?

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version
name: MultiIndex

Comments:
> Would it be faster to use arange too in the line above of my fix instead of Index?

I think you're right, yes
Changed it then
lgtm. will merge / backport on green.
Merged master again to fix benchmarks, will ping on green
@jreback green 
this @phofl 
@meeseeksdev backport 1.2.x
> > Would it be faster to use arange too in the line above of my fix instead of Index?
> 
> I think you're right, yes

https://github.com/pandas-dev/pandas/pull/32477#discussion_r389385303
Interesting, but should not matter?
I don't think this makes a difference, we are always looping over the result as far as I can see.
REGR: diff_2d raising for int8, int16
State: closed

- [x] closes #39050
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Regression
description: Functionality that used to work in a prior pandas version
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
great will merge shortly. the precommit failure fixed elsewhere.
thanks @mzeitlin11 
@meeseeksdev backport 1.2.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 1e900ecbbad92c5db3755a53c2abb21e146921c5
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #39069: REGR: diff_2d raising for int8, int16'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-39069-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #39069 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
TYP: prep _generate_range_overflow_safe for numpy 1.20
State: closed

difficult call, will replace

pandas/core/arrays/_ranges.py:153: error: Incompatible return value type (got "signedinteger[_64Bit]", expected "int")  [return-value]
pandas/core/arrays/_ranges.py:171: error: Incompatible return value type (got "unsignedinteger[_64Bit]", expected "int")  [return-value]

with

pandas/core/arrays/_ranges.py:129: error: Argument 1 to "_generate_range_overflow_safe" has incompatible type "Union[int, number[Any]]"; expected "Union[int, integer[Any]]"  [arg-type]
pandas/core/arrays/_ranges.py:129: error: Argument 2 to "_generate_range_overflow_safe" has incompatible type "Union[int, number[Any]]"; expected "Union[int, integer[Any]]"  [arg-type]
pandas/core/arrays/_ranges.py:137: error: Argument 2 to "_generate_range_overflow_safe" has incompatible type "Union[int, number[Any]]"; expected "Union[int, integer[Any]]"  [arg-type]
pandas/core/arrays/_ranges.py:138: error: Argument 2 to "_generate_range_overflow_safe" has incompatible type "Union[int, number[Any]]"; expected "Union[int, integer[Any]]"  [arg-type]

these are from arithmetic operations on np.integer (including floordiv) which AFAICT should return np.integer and not np.number

can't add the ignores yet, since we will get unused ignore mypy messages - active branch with fixes https://github.com/pandas-dev/pandas/compare/master...simonjayhawkins:numpy-fixes?expand=1

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
closing while we discuss #39513
Regression in replace raising ValueError for bytes object
State: closed

- [x] closes #38900
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Strings
description: String extension data type and string data
name: replace
description: replace method

Comments:
thanks @phofl 
@meeseeksdev backport 1.2.x
Added docs for the change of behavior of isin
State: closed

This is to close issue #38781, I added documentation to explain the change of behavior of the isin function.

- [x] closes #38781 
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Docs

Comments:
Hello @omarafifii! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-01-16 13:11:57 UTC
I added the docs to the isin in the series.py file. Should I also do the same for the one in frame.py?
we may also want to add an entry in the release notes for 1.2 about this change and also a note in the 1.2.1 release notes (other section) that the release notes for 1.2 are updated. 
thanks @omarafifii and @simonjayhawkins 
@meeseeksdev backport 1.2.x
Something went wrong ... Please have  a look at my logs.
TST: skips in extension.test_categorical
State: closed

- [x] closes #38904  
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

cc @simonjayhawkins 

For the searchsorted test, the categorical data starts off ordered as "C" < "A" < "B" but then gets reordered in the test by these lines:

> b, c, a = data_for_sorting
        arr = type(data_for_sorting)._from_sequence([a, b, c])

so the ordering becomes "A" < "B" < "C". This causes failure when "C" is said to be inserted at index 3 (new order) to preserve order rather than the expected 0 (original order).



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Categorical
description: Categorical Data Type

Comments:
thanks @rhshadrach
REF: Move Series.apply/agg into apply
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Will do follow up to share list/dict/str code in Apply for how=apply; they are essentially identical.

Labels:
name: Refactor
description: Internal refactoring of code
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
nice, thanks @rhshadrach
TYP: series apply method adds type hints
State: closed


- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> Thanks @BeanNan for the PR. a couple of suggestions.

Corrected, please take a look
see https://github.com/pandas-dev/pandas/runs/1673701200 for mypy failures
CI has passed, please take a look
can you merge master; your changes will now work in pandas/core/apply.py where thing were moved.
Updated, but there was a slight problem with the CI that doesn't seem to be related to this commit
> Updated, but there was a slight problem with the CI that doesn't seem to be related to this commit

yep, unrelated.