BrandonFors/pandas-github-issues huggingface dataset:
TYP: make typing DatetimeLikeScalar a Union instead of TypeVar
State: closed

In `_typing.py` change `DatetimeLikeScalar` from `TypeVar` to `Union`

Allows removal of 4 `#type: ignore` lines

Per split out from #39501 

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @Dr-Irv
TYP: make EA registry private
State: closed

From #39501 splitting out the part to make the EA `registry` private.  This will make it so that `pyright --verifytypes` accepts pandas public API.




Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @Dr-Irv
REF: move shift logic from BlockManager to DataFrame
State: closed

Partially-fixes one of the currently-disabled ArrayManager tests.

Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
I don't know if it's a particularly useful benchmark, but there is a big slowdown potentially related to this PR: https://pandas.pydata.org/speed/pandas/#frame_methods.Shift.time_shift?python=3.8&Cython=0.29.21&p-axis=1&commits=05a0e98a-bc62e768
best guess is its bc a consolidate is being done by concat, which is called 2x.  easy-ish workaround will be available following #38939
I opened an issue to track my above comment -> https://github.com/pandas-dev/pandas/issues/42246
CLN: Optional[Hashable] in dict type hints
State: closed

Minor clean-up.

```python
>>> from typing import Hashable
>>> isinstance(None, Hashable)
True
```

In old versions of mypy, mypy had a bug and didn't see this, but is has been fixed now.

There are many instances of `Optional[Hashable]` instead of just `Hashable` in the code base, other than in these dicts. I've assumed those should be left as-is for readability and not because it's strictly needed.


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
for a while we had an alias `Label=Optional[Hashable]`, but i think weve been replacing that with just Hashable.  if im right, then you can just go town on all occurrences.  cc @simonjayhawkins ?
> for a while we had an alias `Label=Optional[Hashable]`

that was removed in #39107, although looking back, I didn't do the follow-up

And so I also missed the cases of `Optional[Hashable]` in the codebase that maybe should have been `Label` previously, but there was some discussion on when to use Label https://github.com/pandas-dev/pandas/pull/32371#discussion_r386637945 and it looks like these cases fell through the cracks.

Thanks @topper-123 generally lgtm. I don't think necessary to change the comments but nbd
> if im right, then you can just go town on all occurrences. cc @simonjayhawkins ?

maybe in some cases helps readablity, we have `no_implicit_optional = True` in `setup.cfg` for that reason.
There are many case for function/method parameters like this:

```python
def func_name(kw: Optional[Hashable] = None, kw_2) -> ...:
    ...
```

In such cases the `Optional` is not necessary wrt. mypy, because None is hashable. But I think it has been done this way for readability as `Optional[Hashable]` emphasizes that None is different than other hashables. In the dict cases in this PR, that isn't the case and in these cases None isn't special.

I don't have a very strong opinions, but somewhat prefer `Optional[Hashable]` in cases where None has a different meaning than other hashables to visually show the special status of None.

Thanks @topper-123
STYLE loosen inconsistent namespace check
State: closed

For context, see #40468, where the error which was fixed was arguably a false-positive.

I've loosened the check so that now it only errors / replaces if:
- you imported an object from somewhere in pandas `e.g. from pandas.io.formats import info`
- you also accessed that object as `pd.` or `pandas.` (e.g. `pd.info` or `pandas.info`)

Before, it just checked whether an object appeared both with and without the `pd.` prefix, which was too tight a check - sorry about that :flushed: 

I've also added line number and column offset of the inconsistency for ease of checking/fixing

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
can you merge master; i think the docbuild should be fixed (not sure about the other one)
@MarcoGorelli can you merge master and ping on greenish
@MarcoGorelli if you an merge master once more
@jreback sure, done
thanks @MarcoGorelli
CI: testing wip branch on ci.
State: closed

No need to review. will be closing to clear the queue later today.

Labels:

Comments:
Hello @simonjayhawkins! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/_libs_numba/algos.py`](https://github.com/pandas-dev/pandas/blob/0aa61e06c97f7ddf5143f19c005fee496ff60497/pandas/_libs_numba/algos.py):

> [Line 225:5](https://github.com/pandas-dev/pandas/blob/0aa61e06c97f7ddf5143f19c005fee496ff60497/pandas/_libs_numba/algos.py#L225): [E741](https://duckduckgo.com/?q=pep8%20E741) ambiguous variable name 'l'
> [Line 247:13](https://github.com/pandas-dev/pandas/blob/0aa61e06c97f7ddf5143f19c005fee496ff60497/pandas/_libs_numba/algos.py#L247): [E741](https://duckduckgo.com/?q=pep8%20E741) ambiguous variable name 'l'







































##### Comment last updated at 2021-03-21 13:11:55 UTC
So looks like I can't put off dealing with the numba warnings in order to reduce the noise in the logs.
have added numba to all the envs for testing purposes, so have reopened to re-run tests.

should have now removed the warnings for take_1d in object mode. _pad_2d_inplace still has the object related warnings
DEPR: CategoricalBlock; combine Block.replace methods
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Categorical
description: Categorical Data Type
name: Deprecate
description: Functionality to remove in pandas

Comments:
we're gonna get down to 3 Block classes or die trying
sgtm!
CLN: remove FloatBlock, share _can_hold_na
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
great
TYP: io.sas
State: closed

The way we pass state back and forth is a bit nutty, punting on that.

Labels:
name: IO SAS
description: SAS: read_sas
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
imagine these casts can be removed if the some of the reading functions return properly bytes?
> imagine these casts can be removed if the some of the reading functions return properly bytes?

yah i expect itll get resolved before too long
DOC/TYP: index.take return val
State: closed

xref https://github.com/pandas-dev/pandas/pull/40513#discussion_r597541594

Labels:
name: Docs
name: Index
description: Related to the Index class or subclasses
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@jbrockmendel can you take another look, @jreback comments have been addressed.
Thanks @mzeitlin11
DOC fix misspelling of countries
State: closed

simple misspelling fix. From "counties" to "countries"
- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:

Comments:
Thanks for the PR, but this is correct (there is a "Dade" county in Florida for example). Please feel free to take any other issue labeled `good first issue`
DataFrame.rolling documentation of method='table'
State: closed

- [X] closes #40374 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


Labels:
name: Docs
name: Window
description: rolling, ewma, expanding

Comments:
Hello @joshlk! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/core/window/rolling.py`](https://github.com/pandas-dev/pandas/blob/3f1b20f9298a5eec0cb7256f054b263d79a182db/pandas/core/window/rolling.py):

> [Line 1615:9](https://github.com/pandas-dev/pandas/blob/3f1b20f9298a5eec0cb7256f054b263d79a182db/pandas/core/window/rolling.py#L1615): [E122](https://duckduckgo.com/?q=pep8%20E122) continuation line missing indentation or outdented
> [Line 1619:1](https://github.com/pandas-dev/pandas/blob/3f1b20f9298a5eec0cb7256f054b263d79a182db/pandas/core/window/rolling.py#L1619): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 1628:1](https://github.com/pandas-dev/pandas/blob/3f1b20f9298a5eec0cb7256f054b263d79a182db/pandas/core/window/rolling.py#L1628): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 1638:1](https://github.com/pandas-dev/pandas/blob/3f1b20f9298a5eec0cb7256f054b263d79a182db/pandas/core/window/rolling.py#L1638): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace



##### Comment last updated at 2021-04-14 11:10:21 UTC
@joshlk ci is failing can you fix-up
> @joshlk ci is failing can you fix-up

Hi, I've tried adding the docs to the apply doc-string but I'm unsure how to do it - my latest commit isn't working. Sorry I am unfamiliar with how the doc-string is being generated. Could you please help
you can run the docstring validation script locally. (if on windows use git bash or wsl)

to build the docs, see https://pandas.pydata.org/pandas-docs/dev/development/contributing_documentation.html#building-the-documentation

this PR is also failing pre-commit checks (black and trailing whitespace) see https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit
@joshlk can you have a look at the points of the previous comment, and try to leave the CI green please?
Thanks for the PR, but appears that progress has stalled. Closing, but let us know if you're still interested in working on this and we'd be happy to reopen.
STYLE no no-string-hints, combine some hooks
State: closed

I got a PR into [pyupgrade](https://github.com/asottile/pyupgrade/blob/c90a455fceacdecc7484dab8519688353b6b2931/pyupgrade/_plugins/typing_pep563.py) to do what no-string-hints did (and in a much better way, their maintainer is amazing), so my little tool can be archived / removed from here :tada: 

Also combining some other hooks while I'm here

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @MarcoGorelli
CLN/PERF: remove catching of numpy deprecation warning in comparison_op
State: closed

xref https://github.com/pandas-dev/pandas/issues/39146

In some of the `FrameWithFrameWide` benchmarks, this takes up around 15% of the time. 

While I don't think this should be needed (but let's see if warnings appear in the CI builds).

Labels:
name: Performance
description: Memory or execution speed performance
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Clean

Comments:
@jbrockmendel this warning filter was added in https://github.com/pandas-dev/pandas/pull/32779. AFAIU, the reason the warning turned up in that PR is because you started to perform ops blockwise (using a 2D numpy datetime64[ns] array), instead going through Series op (where we use the EA). 

`arithmetic_op` already calls `ensure_wrapped_if_datetimelike` on both left/right operands, so this makes `comparison_op` consistent with that (I know I would like to move out `ensure_wrapped_if_datetimelike` alltogether, but can be dealt with together with the other occurences later).
CLN/PERF: remove unused out kwd in take_nd
State: closed

And in a few places where we know allow_fill=False, just call the numpy version directly.

Labels:
name: Performance
description: Memory or execution speed performance
name: Clean

Comments:
lgtm. any perf benefit?
> any perf benefit?

mostly in the places where we call ndarray.take directly instead of going through algos.take
@jbrockmendel did you run benchmarks for this change? 
There are lots of regressions at https://pandas.pydata.org/speed/pandas/#regressions?sort=1&dir=desc that point to this commit (although it's not always very accurate in pointing to the correct commit, so it can also be something else, but so should be checked)
i profiled with ipython, didnt run the asv suite.  do you have a hypothesis for what part of this might be the culprit?
>  do you have a hypothesis for what part of this might be the culprit?

No, I didn't check it in detail. This PR is touching several places, and it is also a variety of benchmarks that seem to be affected (reindexing, merge, groupby, ops, select_dtypes, dropna)
@jbrockmendel were you able to look into this? 
(if not, I would propose to revert it for now (because that will only get harder later), and add it again in smaller parts)
> (if not, I would propose to revert it for now (because that will only get harder later), and add it again in smaller parts)

OK i guess
@jbrockmendel can you follow-up on this?
BUG: fillna overwriting other missing vals
State: closed

- [x] closes #40498
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
> i am not sure i buy that this is a bug at all.
> 
> ```
> In [149]: series.isna()                                                                                                                                               
> Out[149]: 
> 1    False
> 2     True
> 3     True
> 4    False
> dtype: bool
> ```
> 
> Trying to use mixed null values in an `object` series is just not supported.

I think the confusion from #40498 is not that `isna(None) = True`, but that the value at index 2 is modified even though the argument `{'3':0}` should ensure that only index 3 is modified. 
> > i am not sure i buy that this is a bug at all.
> > ```
> > In [149]: series.isna()                                                                                                                                               
> > Out[149]: 
> > 1    False
> > 2     True
> > 3     True
> > 4    False
> > dtype: bool
> > ```
> > 
> > 
> > Trying to use mixed null values in an `object` series is just not supported.
> 
> I think the confusion from #40498 is not that `isna(None) = True`, but that the value at index 2 is modified even though the argument `{'3':0}` should ensure that only index 3 is modified.

I am ok with that, but your patch is not structured that way. it shouldn't be testing for nulls
Sorry, can you please clarify what you mean by that? 

What the patch should be doing is fixing the following: for the series `ser = Series({'1': 1,'2': None,'3': np.nan, '4': 1})`, when the fill argument '{'3':0}' hits the line I changed, it previously became `[NaN, NaN, 0, NaN]`. 

When this hits `ObjectBlock.fillna`, mask is determined from `isna(ser)`, giving `[False, True, True, False]`. So then when `putmask_inplace` is called with this mask, `None` is replaced with `np.nan` since the mask is `True` at the position.

To avoid having to change the behavior of lower level `fillna` functionality, this patch changes the fill argument to instead become `[NaN, None, 0, NaN]`, so that `None` just ends up being replaced by itself.
TYP: parsers.pyi
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
can you merge master
rebased + green
BUG: Respect ignore_index in Series.explode
State: closed

- [x] closes https://github.com/pandas-dev/pandas/issues/40487
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thanks @dsaxton
DOC: Fix documentation for DataFrame.groupby.transform
State: closed

 - The doc template should no longer reference "Series" transforms explicitly.
 - Make consistent the formatting for references to function argument names.

Labels:
name: Docs
name: Groupby
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
looks fine. @rhshadrach if any comments.
Thanks @vladu!
Thanks, Richard. I appreciate your feedback. This was my first PR to pandas, and I was trying to do something low impact / low controversy. Will be more comprehensive going forward. 

-- Vlad


@vladu - I wouldn't recommend changing a thing. To make sure - my complaint about the current docs for `DataFrame.transform` was very much outside the scope of this PR. This was well done.
BUG: Remove artificial precision limit in rolling var & std
State: closed

- [x] closes #37051, xref #40448, xref #39872
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Window
description: rolling, ewma, expanding

Comments:
thanks @mroeschke
TYP: nattype.pyi
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
updated  + green
PERF: increase the minimum number of elements to use numexpr for ops from 1e4 to 1e6
State: closed

See https://github.com/pandas-dev/pandas/issues/40500 for the analysis.

Labels:
name: Performance
description: Memory or execution speed performance
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
prob need to update the docs a bit here https://pandas.pydata.org/docs/user_guide/enhancingperf.html
> prob need to update the docs a bit here https://pandas.pydata.org/docs/user_guide/enhancingperf.html

That page is about using numexpr in `eval`/`query`, so that is independent from this change. On the basic page is a section about accelerated ops: https://pandas.pydata.org/docs/user_guide/basics.html?highlight=numexpr#accelerated-operations 
But it doesn't mention the minimum number of items, so that doesn't need to be updated (we should update the table id at some point switching to the ArrayManager, as that will be influenced, but so that's still for the future)
Looks like this wasnt green when merged, could be responsible for broken CI?
PERF: optimize is_numeric_v_string_like
State: closed

In some of the arithmetic benchmarks (xref https://github.com/pandas-dev/pandas/issues/39146#issuecomment-799482699), just this `is_numeric_v_string_like` check takes up 15-35% of the overall time. 

This improves the performance of this check by using some ndarray-specialized dtype checks (checking the kind instead of the generic functions):

```
In [1]: from pandas.core.dtypes.common import is_numeric_v_string_like

In [2]: arr = np.array([1, 2, 3])

In [3]: %timeit is_numeric_v_string_like(arr, 2.0)
2.3 µs ± 46.9 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- master
482 ns ± 41.8 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
nice!  two questions, neither blockers for this PR

1) IIUC this check exists to prevent a numpy-issued deprecation/future warning.  Do we know when that ceases to be an issue?

2)  Looks like is_string_like_dtype is only used in one other place (in dtypes.missing).  lets just rip it out
> Looks like is_string_like_dtype is only used in one other place (in dtypes.missing). lets just rip it out

Or I could update the helper function to what I used here, and then keep using it in those 3 places?

> Or I could update the helper function to what I used here, and then keep using it in those 3 places?

either way i guess.  i prefer to inline the check since i frequently find myself having to go look up what the is_foo_dtype's exact behavior is
OK, I am certainly fine with inlining here
> IIUC this check exists to prevent a numpy-issued deprecation/future warning. Do we know when that ceases to be an issue?

Not directly an idea, but at least with the latest numpy release, you still get this wrong behaviour:

```
In [1]: np.array([1, 2]) == "a"
<ipython-input-1-c406769a5f40>:1: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  np.array([1, 2]) == "a"
Out[1]: False

In [2]: np.__version__
Out[2]: '1.20.1'
```

so even if that would change in a next release, we are still going to need to workaround it for quite a while ..
[ArrayManager] TST: include subset of ArrayManager tests in all CI builds
State: closed

xref #39146


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
@jreback @jbrockmendel as discussed, I am trying here the idea to run a specific subet of tests most relevant for the ArrayManager on each of the builds (right now testing with `/frame/` tests). 

For that, I added an "arraymanager" mark, and then doing a second pytest run with this mark activated. 

(there are still some actual failures discovered by this to look at / solve)
is the test_median failure a false-positive?
No, that was an actual failure (forgot a colon in the `filterwarnings` signature)
This is all green now. 

For now, it only adds the /frame/ tests, we can later add the mark to more tests if we want. With only the frame tests, this currently adds eg 135s in the "actions-38.yaml, not slow and not network and not clipboard" build (so 2min to an overall of 42min), which I think is fine.
> so 2min to an overall of 42min), which I think is fine

Agreed.  Do you expect to keep adding to this though?
>  Do you expect to keep adding to this though?

Some I suppose, but I think we can evaluate that one at a time when actually adding it whether it adds too much time or not. 
(eg the full `pandas/tests/indexing` takes less than a minute)

looks good, can you rebase once more just to make sure (merged a bunch of things)
Rebased, and all green.
@jorisvandenbossche is this the source of the codecov CI failure?  looks like it wasnt green when merged, but the codecov failure may have predated this.
>  looks like it wasnt green when merged, but the codecov failure may have predated this.

Ah, yes, sorry, I generally ignore codecov failures since they have been so flaky in the past (but maybe that's better now), but I missed that it wasn't the "patch" coverage that was failing, but the "project" one.

I suppose just removing the coverage from the ArrayManager pytest invocation should fix it. Will do a PR.

https://github.com/pandas-dev/pandas/pull/40711