BrandonFors/pandas-github-issues huggingface dataset:
REGR: Series[numeric] comparison with str raising on numexpr path
State: closed

- [x] closes #36377, closes #35700
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
great, hopefully this backports cleanly
@MeeseeksDev backport to 1.1.x
DOC: read_excel skiprows documentation matches read_csv (#36435)
State: closed

- [x] closes #36435 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Updated `read_excel` documentation to match `read_csv` for the `skiprows` parameter.
Changed `test_read_excel_skiprows_list` to `test_read_excel_skiprows`: now also tests `read_excel` with an `int` and a `callable`. 

Labels:
name: Docs
name: IO Excel
description: read_excel, to_excel

Comments:
thanks @ahgamut .  Nice first contribution!
REF: pandas/io/formats/format.py
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Partially addresses https://github.com/pandas-dev/pandas/issues/36407

Before splitting ``DataFrameFormatter`` into more dedicated classes, I decided to refactor the class itself, to make the outstanding refactoring more manageable.

As suggested by @jreback, I made this refactor small, trying to split big functions into smaller ones and find some better naming.
Yet, rather small changes are done so far, just to keep diffs readable.
Once approved, I will move on to further refactor.


Labels:
name: Refactor
description: Internal refactoring of code
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
Hello @ivanovmg! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-19 18:09:57 UTC
Thanks @ivanovmg for the PR. General Question.

DataFrameFormatter is not public. Is the validation always necessary or only during \_\_init__? Would _validator_foo() validators (perhaps even in attrs style https://www.attrs.org/en/stable/examples.html#validators) be more appropriate than property setters.
> 
> 
> Thanks @ivanovmg for the PR. General Question.
> 
> DataFrameFormatter is not public. Is the validation always necessary or only during __init__? Would _validator_foo() validators (perhaps even in attrs style https://www.attrs.org/en/stable/examples.html#validators) be more appropriate than property setters.

My intention was indeed to extract parameters validation/conversion. It is not only validation. I will need to carefully look through the code to see if there is any assignment to the attributes via the setters defined. Probably these setters are used only once here.

I think that if we potentially derive ``ConsoleFormatter``, ``HTMLFormatter`` and ``LatexFormatter`` from ``DataFrameFormatter``, these properties with setters will be useful, since they will make initialization neat (just self.some_attr = some_attr). On the other hand it may be not very clear that there is a setter hiding behind.
I couldn't help but extracting some more methods. I hope the diffs are is still readable. These changes will be helpful for the upcoming polymorphism updates (formatters with and without truncation).
Something is wrong with the CI. The environment is not getting built time after time.
we already have a cached property decorator

cache_readonly

though only use it if we are actually doing something non trivial 
yeah cache_readonly is in cython without a stub file, so using here would undo all the typing benefits since properties would resolve to 'Any'
thanks @ivanovmg keep em coming!
[DOC]: Add warning about rolling sums with large values
State: closed

- [x] closes #11645 
- [x] closes #28244 (probably closes that one too)
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Added a warning within the user guide.


Labels:
name: Docs
name: Window
description: rolling, ewma, expanding

Comments:
thanks @phofl very nice! keep em coming!
PERF: construct DataFrame with string array and dtype=str
State: closed

Avoid inefficient call to `arr.astype()` when dtype is `str`, and use ensure_string_array instead.

Performance example:

```python
>>> x = np.array([str(u) for u in range(1_000_000)], dtype=object).reshape(500_000, 2)
>>> %timeit pd.DataFrame(x, dtype=str)
391 ms ± 17.7 ms per loop  # master
11.9 ms ± 131 µs per loop  # after this PR
```

xref #35519, #36304 & #36317.


Labels:
name: Performance
description: Memory or execution speed performance
name: Strings
description: String extension data type and string data

Comments:
Nice! Add a benchmark for it? 
And is this speed-up only for constructing from an array, or also eg from dict `{'col': x}` ?
> And is this speed-up only for constructing from an array, or also eg from dict {'col': x} ?

This PR just affects the path for multi-dimensional arrays, while `{'col': x}` takes the 1-dim path, so not affected by this. The one dim-path was however helped by #36317.

> Nice! Add a benchmark for it?

Yeah, I'll add it after the travis checks work again.

Updated and added ASVs.
thanks very nice!
BUG: get_indexer returned dtype
State: closed

- [x] closes #36359
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Changed the return dtype of IndexEngine’s `get_indexer()` and `get_indexer_non_unique()`. I decided to not include tests because dtypes are already implicitly tested in the following:

- pandas/tests/indexes/test_base.py (TestIndex.test_get_indexer)
- pandas/tests/base/test_misc.py (test_get_indexer_non_unique_dtype_mismatch)

Also, I ran the asv benchmarks:

```
asv continuous -f 1.1 upstream/master HEAD -b ^indexing_engines
BENCHMARKS NOT SIGNIFICANTLY CHANGED.
```


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
cc @jbrockmendel 
LGTM
thanks @alexhlim
Update isort version in pre-commit config
State: closed

Previously, a mismatch between the isort versions specified by
pre-commit and by the files used on CI made it impossible to make a
commit which passed both pre-commit and CI lints.

See #36426 . This is not a complete solution as the flake8 version differences are intentional and hopefully temporary (see https://github.com/pandas-dev/pandas/pull/36412#issuecomment-694072443 ), but it should resolve the current breakage.

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @clbarnes
fix inconsistent index naming with union/intersect #35847
State: closed

- [X] closes #35847
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This takes care of some inconsistency in how names are handled by `Index` functions `union` and `intersection`, as discussed in #35847. I believe this covers all index types, either through the base class or in the subclass when necessary.

What I've implemented here actually uses the unanimous convention, wherein all input names must match to get assigned to the output. Originally, I was thinking consensus would be better (assign if there is only one non-None input name), but looking through the existing tests, it seems that unanimous was usually expected. I also had some worries about whether index names would become too "contagious" with consensus. Anyway, it's easy to change between the two if people have strong opinions on this.

Labels:
name: Index
description: Related to the Index class or subclasses

Comments:
Ok, added in a code example to whatsnew showing the new/consistent naming behavior when aggregating. Also split off those new tests.

Regarding copying in the various empty/equality cases, copy here means a deep copy including values, right? Seems like some of the tests are using "is" which breaks when you add in copying. I can look into that and the nan issues in a subsequent PR if that works.
@jreback fixed the docs indentation issue, that's green now. i'm having trouble figuring out went wrong with some of the other tests. is it some kind of internal CI error?
CI failures were unrelated, should be fixed if you merge master
I think we're mostly good here, but the travis-ci build for ARM just kinda bailed without comment.
Thanks @jbrockmendel! Pushed all your suggested changes.
@iamlemec can you merge master; IIRC this was pretty close to ready
@jbrockmendel sure thing, just pushed a rebase onto master
thanks @iamlemec very nice!
thanks for all the help @jreback and @jbrockmendel!
Bump flake8 version in pre-commit-config.yaml
State: closed

The flake8 version used in pre-commit-config.yaml seems to emit spurious errors when there are mypy comments for ignoring specific kinds of type errors. Bumping to the latest version of flake8 seems to fix this.

```
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

pandas/core/generic.py:701:36: F821 undefined name 'ignore'
pandas/core/generic.py:701:36: F821 undefined name 'arg'
pandas/core/generic.py:4220:39: F723 syntax error in type comment 'ignore[return-value, arg-type]'
pandas/core/generic.py:4283:39: F723 syntax error in type comment 'ignore[return-value, arg-type]'
```

For reference this is line 701 that it's complaining about:
```python
            new_values, *new_axes  # type: ignore[arg-type]
```

ref https://github.com/PyCQA/pyflakes/pull/455

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
So we are actually pinning flake8 to an older version (https://github.com/pandas-dev/pandas/issues/34150) for the code checks. But it seems that most issues have been resolved (eg https://github.com/pandas-dev/pandas/pull/34152), only the flake8-rst compatibility not yet. But that is something we don't need to care about for pre-commit. So updating here should be good.
CLN: sql static method
State: closed

- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
Not sure why my commits from #35224 are included in this PR. Maybe because I changed my email in config user.email. But the changes I made for this PR are in `sql.py`

> Not sure why my commits from #35224 are included in this PR. Maybe because I changed my email in config user.email. But the changes I made for this PR are in `sql.py`

I'd guess it's because you created a new branch from your previous one and then added new commits on top of it, instead of resetting to upstream/master before starting new work.

You can get rid of unwanted commits with `git rebase -i upstream/master` and then dropping any commits you don't want (note that you'll then need to use the `-f` flag when pushing)
> > Not sure why my commits from #35224 are included in this PR. Maybe because I changed my email in config user.email. But the changes I made for this PR are in `sql.py`
> 
> I'd guess it's because you created a new branch from your previous one and then added new commits on top of it, instead of resetting to upstream/master before starting new work.
> 
> You can get rid of unwanted commits with `git rebase -i upstream/master` and then dropping any commits you don't want (note that you'll then need to use the `-f` flag when pushing)

Thanks, I can do that, but the changes now look correct, and it does not conflict with the base branch, it's fine like this as well I guess?
I think it should be fine, yes :) While you're here modifying these files, can you also add type annotations to the signatures you've changed (where possible)?
BUG: Categorical.sort_values inplace breaking views
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Categorical
description: Categorical Data Type

Comments:
this doesn't have any user facing appearances rigth?
> this doesn't have any user facing appearances rigth?

Only if a user was using Categorical.sort_values directly
TYP: alias IndexLabel without Optional 
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Previously in ``_typing.py`` we had the alias ``IndexLabel``, which contained ``Optional``.
The present PR eliminates ``Optional`` from the alias, but adds it in the code where necessary.

Related to the issue raised by @simonjayhawkins in https://github.com/pandas-dev/pandas/pull/36046#discussion_r485510442

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Travis CI failed at miniconda build. Can anyone restart?
Since `IndexLabel` can be a `Label`, and `Label = Optional[Hashable]`, isn't `Optional[IndexLabel]` the same as `IndexLabel`?
lgtm. @simonjayhawkins can you address @rhshadrach question.
> Since `IndexLabel` can be a `Label`, and `Label = Optional[Hashable]`, isn't `Optional[IndexLabel]` the same as `IndexLabel`?

for typing, yes. just as Union[int, float] is equivalent to float and the Optional in function signatures could be implicit. We do this for clarity and consistency.

However, the actual convention that we should eventually adopt maybe determined by the outcome of #28585 and #33025.

If we go the route of showing the aliases in the docs ( https://github.com/scikit-learn/scikit-learn/issues/16705#issuecomment-683717061 and https://github.com/numpy/numpydoc/issues/196 related), having the Optional maybe preferable.
thanks @ivanovmg
CLN: remove trailing commas for black update
State: closed

Working on: CLN remove unnecessary trailing commas to get ready for new version of black #35925.
Checked the following files to make sure they are ready for black :
 pandas/tests/scalar/timestamp/test_arithmetic.py
 pandas/tests/scalar/timestamp/test_constructors.py
 pandas/tests/series/methods/test_argsort.py
 pandas/tests/series/methods/test_convert_dtypes.py
 pandas/tests/series/methods/test_drop_duplicates.py
 pandas/tests/series/methods/test_interpolate.py
 pandas/tests/series/methods/test_unstack.py
 pandas/tests/series/test_cumulative.py
 pandas/tests/test_algos.py

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Clean

Comments:
Thanks @cjlynch278 !
Backport PR #36371 on branch 1.1.x (BUG: Fix MultiIndex column stacking with dupe names)
State: closed

Backport PR #36371: BUG: Fix MultiIndex column stacking with dupe names

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex

Comments:
The failure is also on 1.1.x branch. https://travis-ci.org/github/pandas-dev/pandas/jobs/728460104

so merging otherwise future autobackports could fail with merge conflicts on whatsnew
BLD/CI: fix py39 ci #36296
State: closed

closes  #36296 

Need  #36298 first.

Labels:
name: Build
description: Library building on various platforms

Comments:
go ahead and add the commit from #36298  here
ok can u add a note in 1.1.3 whats new that we support 3.9
also maybe need small edits in setup.py

and add a note that need cython==0.29.21 in the whats new as well
cc @pandas-dev/pandas-core 

backporting this would bump the required cython for 1.1.3

alt we could release this as 1.2
(and bump current to 1.3)

i am ok with doing this to support 3.9 

as this is not a runtime dep changing


Is Cython required for sdist builds or only for git repo builds?  
> Is Cython required for sdist builds or only for git repo builds?

It's required for sdists, but we set it as a requirement in pyproject.toml so that users don't need it in their env ahead of time.
I'm fine with bumping the Cython requirement. We just need to update it in a few places (setup.py, pyproject.toml, install.rst, ...)
@VirosaLi can you update the cython requirements in these places as well.

> (setup.py, pyproject.toml, install.rst, ...)
@TomAugspurger @simonjayhawkins if you'd have a quick look.
I doubt this will autobackport because of differences in the ci setup. so I think OK to merge and then patch as needed if issues on 1.1.x or MacPython.
> I doubt this will autobackport because of differences in the ci setup. so I think OK to merge and then patch as needed if issues on 1.1.x or MacPython.

@simonjayhawkins yep figured that.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 11d5fc9e89ad8e238ece42890482bae875d9de98
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #36393: BLD/CI: fix py39 ci #36296'
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-36393-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #36393 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
thanks @VirosaLi really nice!
@VirosaLi did you want to do the backport?
> @VirosaLi did you want to do the backport?

Sure, I'll do that later today.
wheel build failing https://dev.azure.com/pandas-dev/pandas-wheels/_build/results?buildId=42575&view=logs&j=a846d25a-e32c-5640-1b53-e815fab94407
[TST]: Groupby raised error with duplicate column names
State: closed

- [x] closes #31735 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Added a test to avoid regression

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby

Comments:
Hello @phofl! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-15 20:42:49 UTC
thanks @phofl
BUG: Always cast to Categorical in lexsort_indexer
State: closed

- [x] closes #36383
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Categorical
description: Categorical Data Type

Comments:
Not sure if this is related, but I noticed a problem in Categorical.sort_values setting self._codes = foo instead of self._codes[:] = foo
thanks @dsaxton 


@jreback you milestone 1.2. change note in 1.1.3. which should it be?
@jreback ok to backport? see https://github.com/pandas-dev/pandas/pull/36385#issuecomment-694275310
yeah this is ok
thanks
@meeseeksdev backport 1.1.x
REF: _is_compatible_with_other -> _can_union_without_object_cast
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Changes the behavior in one test where ATM we behave differently for Series/ndarray than we do for Index arg

Labels:
name: Refactor
description: Internal refactoring of code
name: Index
description: Related to the Index class or subclasses

Comments:
> Changes the behavior in one test where ATM we behave differently for Series/ndarray than we do for Index arg

Can you show an example of that behaviour change?
> Can you show an example of that behaviour change?

https://github.com/pandas-dev/pandas/pull/36384/files#diff-a3619540517f93ee7c657a439b986962L50

```
everything = tm.makeDateIndex(10)
first = everything[:5]
second = everything[5:]

expected_old = first.astype("O").union(pd.Index(second.values, dtype="O")).astype("O")
expected_new = first.union(second)

case = list(second.values)  # <-- same for np.array(second.values)
result = first.union(case)
tm.assert_index_equal(result, expected)
```

ATM `_is_compatible_with_other` checks specifically for Index subclasses as opposed to dtype-based checks, and so we end up converting to object when we have anything that isn't an Index.
can rebase
Rebased +green
> ATM _is_compatible_with_other checks specifically for Index subclasses as opposed to dtype-based checks, and so we end up converting to object when we have anything that isn't an Index.

Thanks, then this is totally a reasonable change ;) 
Can you add a whatsnew note about it? I would call it a bug fix.
very nice!
ADMIN: Update stale PR action
State: closed

I think also that it could be good to skip the PR message for when debugging is turned off and only label / manually review the labeled PRs to start (until we're comfortable with the tagging logic and message)

Labels:
name: Admin
description: Administrative tasks related to the pandas project

Comments:
restarting ci

```
=========================== short test summary info ===========================
FAILED pandas/tests/test_sorting.py::TestSafeSort::test_codes_out_of_bound[-1]
= 1 failed, 75231 passed, 1809 skipped, 1180 xfailed, 6 xpassed, 69 warnings in 1502.68s (0:25:02) =
```
@simonjayhawkins Merging this in as it's still in debug mode. I'm finding the labeling to be pretty helpful so would be curious to get your thoughts on turning this on.
DOC: move release note for #36175 (pt1)
State: closed

xref https://github.com/pandas-dev/pandas/pull/36363/files#r488227748

Labels:
name: Docs

Comments:
@simonjayhawkins future note: you could also do it in a single PR (targetting master), and update the diff (to remove the changed in the 1.2 file) when doing the backport. 
(I personally think that is cleaner/clearer, although then meeseekdev won't be able to automate the backport of course ..)
CLN: Numba internal routines
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

- Remove some leftover implementation after https://github.com/pandas-dev/pandas/pull/36240
- Delegate numba func caching to methods in `numba_.py`
- Combine some helper methods

Labels:
name: Clean

Comments:
very nice!
BUG: FooIndex.insert casting datetimelike NAs incorrectly
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry



Labels:
name: Datetime
description: Datetime data dtype
name: Interval
description: Interval data type

Comments:
any perf issues? can you rebase
> any perf issues?

we'll take a hit in cases where we now cast to object, but id call those bugs in master.

> can you rebase

done + green