BrandonFors/pandas-github-issues huggingface dataset:
DEPR: ExtensionOpsMixin -> OpsMixin
State: closed

- [x] closes #37080
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Deprecate
description: Functionality to remove in pandas
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
thanks @jbrockmendel 
Sorry, but this was not really ready:

- There has been no discussion about what the actual use case of `ExtensionScalarOpsMixin` (the issue mentions we don't use it internally, but it's goal is to serve external users, not internal)
- There is no indication in this PR of how to replace `ExtensionScalarOpsMixin` as a user of it
- The warning message suggests `OpsMixin` (but which is not actually a replacement), but this is not publicly exposed

I don't know if we should keep the class, but we should at least discuss it (and check if there is usage of it), and if we deprecate it give a clear reasoning and alternative.
@jorisvandenbossche ahh re-read your comment on the issue. if you want to revert ok, but to be honest this looks better / simpler.
cc @Dr-Irv, who I believe implemented the mixin. I suspect you're using it?
I would say, let's revert it for now, until we have better discussed this. @jbrockmendel can you do that?
Actually, clicking the button is not that hard ;) -> https://github.com/pandas-dev/pandas/pull/38158
I reopened the issue, let's continue the discussion there: https://github.com/pandas-dev/pandas/issues/37080
> Actually, clicking the button is not that hard ;) -> https://github.com/pandas-dev/pandas/pull/38158

actually you can do this from the UI itself? (create a revert)
BUG: do not stringify file-like objects
State: closed

- [x] closes #38125
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

I will finish this PR (fix the mypy issues) after #38018 is merged.


Labels:
name: Bug
name: IO Data
description: IO issues that don't fit into a more specific label
name: IO CSV
description: read_csv, to_csv

Comments:
can you merge master as merged the PathLike change
@jreback ping
thanks @twoertwein 
@meeseeksdev backport 1.2.x
test backportability of #38120
State: closed

#38120

Labels:
name: Frequency
description: DateOffsets
name: MultiIndex
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
@jbrockmendel the test faiures are for PeriodArray/Index. There must be some differences on master on how this is handled. will continue to investigate.
so it appears the class hierarchy has changed.

I think the docstring for DatelikeOps, 'Common ops for DatetimeIndex/PeriodIndex, but not TimedeltaIndex.' and TimelikeOps is now confusing.

I'm not sure why the merge put the new implementation of factorize in `DatetimeLikeArrayMixin` instead of `TimelikeOps`, but if it's put in `TimelikeOps` then the base EA implementation of factorize will be higher in the MRO.

so have kept it in `DatetimeLikeArrayMixin` and overridden in `PeriodArray`
BUG: Categorical[dt64tz].to_numpy() losing tz
State: closed

- [x] closes #26406
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Categorical
description: Categorical Data Type

Comments:
ok for 1.2, can you add a whatsnew note.
whatsnew added + greenish
DOC: add contibutors to 1.2.0 release notes
State: closed


Labels:
name: Docs

Comments:
test failure unrelated

```
=========================== short test summary info ============================
FAILED pandas/tests/io/test_gcs.py::test_to_csv_compression_encoding_gcs[zip-cp1251]
= 1 failed, 144014 passed, 3150 skipped, 1431 xfailed, 31 warnings in 3195.95s (0:53:15) =
```
REF: use np.where instead of maybe_upcast_putmask in nanops
State: closed

This way we make at most one copy.  Also we get rid of the only use of maybe_upcast_putmask that has fill_value of anything other than np.nan, so in a follow-up it can be simplified.

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
great never like this `maybe_upcast_putmask`, was in place for older numpies.
good because more of it is getting deleted soon
ENH: Implement dict-like support for rename and set_names in MultiIndex
State: closed

- [x] closes #20421
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

I tried my hand in adding support for dict-like renamings for MultiIndexes. This seemed as the most straight forward way to me, because we do not have to add complex logic as long as we can cast the dict-like objects to regular configurations.

We probably must add a docstring to MultiIndex.rename now?

Labels:
name: Enhancement
name: MultiIndex

Comments:
cc @Dr-Irv if you'd have a look. this is api you were expecting (from the OP)?
> cc @Dr-Irv if you'd have a look. this is api you were expecting (from the OP)?

Yes, that is the API.  I didn't check the implementation, but the tests should reflect the "normal" use case of having a `MultiIndex` with names that are different for each level, and you want to change one or two of the names.  Also, should check that behavior is consistent with `DataFrame.rename()` if a name in the `dict` argument is not one of the level names - I'm not sure what we do in `DataFrame.rename()` if that is the case.
@Dr-Irv 

Concerning missing names: I have oriented me on df.rename, as you also suggested. df.rename does nothing for missing index elements or column names.

Additionally for duplicates: Same holds true here. Is also done like df.rename. It produces duplicates if we have the same name twice in the original and it also produces duplicates if the targets already exist.
ping
@jreback green
thanks @phofl
BUG: Index.intersection casting to object instead of numeric
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Index
description: Related to the Index class or subclasses

Comments:
already has whatsnew.  mind if i add the copy=False in the next pass?  lots in the pipeline
> already has whatsnew. mind if i add the copy=False in the next pass? lots in the pipeline

sure
I don't know if it was this PR (from this range https://github.com/pandas-dev/pandas/compare/65319af6...f04ec866), but the `intersection` method got a bit slowdown in case of numeric indexes: https://pandas.pydata.org/speed/pandas/#index_object.SetOperations.time_operation?p-dtype='int'&p-method='intersection'
Ah, might also be https://github.com/pandas-dev/pandas/pull/36927
API: preserve freq in DTI/TDI.factorize
State: closed

cc @simonjayhawkins 
closes #35563 

this is an old branch I resurrected and rebased, so has some excess cruft that would need to be removed if we want to move forward.

LMK if there are any tests you'd like to see added based on the other issue (or just go ahead and push them if you like)

Labels:
name: Datetime
description: Datetime data dtype
name: Frequency
description: DateOffsets
name: MultiIndex
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
thanks @jbrockmendel I'll look in detail tomorrow and add tests as required.
this is going to be hard to backport
> if you can get this passing on the backport as well ok to merge, but pls put up a separate patch for that to see if its even possible.

sure
> this is going to be hard to backport

maybe doable, in https://github.com/pandas-dev/pandas/pull/38137, the ci has not completed yet but looks like tests are passing (except for np-dev which is unrelated and failing on other PRs)
yeah ok with just making this work here, can fixup later / cleanup
@jreback it would be good to get this into master before 1.2.0rc0 if it's ready. not critical but would only need one backport. backporting to 1.1.5 does not block 1.2.0rc0

if more discussion needed, fine. no need to be under any pressure to merge.
@simonjayhawkins no i think this is fine. I am sure that @jbrockmendel will have a cleanup on top of this, but good for now. thanks.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 c29c17630d73b834adcfa97837db01f43ff06ce8
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #38120: API: preserve freq in DTI/TDI.factorize'
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-38120-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #38120 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
ENH: include conversion to nullable float in convert_dtypes()
State: closed

xref https://github.com/pandas-dev/pandas/issues/38110

There is one potentially corner case: what with floats that are all "integer"-like? I think we want to keep returning nullable int for that, at least by default, and that is what I did now in this PR But we might want to add a parameter controlling that behaviour? (but that can also be added later on if there is demand for it)

Labels:
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
~~Also, note, this builds upon https://github.com/pandas-dev/pandas/pull/38113. Once that is merged, the changes I needed to do to the test cases will become clear.~~ https://github.com/pandas-dev/pandas/pull/38113 is merged
> liking less and less the fact that we have many options here.
>
>would change to an include=, exclude interface. (might involve deprecating the existing options). am -1 on doing this for 1.2 for this reason.

We had a long discussion about this in the original PR (https://github.com/pandas-dev/pandas/pull/30929), and I want to remind that it was actually you who asked for the separate options .. (I personally think we should not have added ay of those options in the first place)

If we want to change the interface of `convert_dtypes`, we can do that later as well. I don't think that needs to block this PR. I could also leave out the option for now (and only have the default behaviour of True)

and am happy to revisit but not at the 11hour 


I am happy with leaving out the option, but I think we should still include the actual functionality in 1.2. The float EA array already is included in 1.2, this just enables it in the `convert_dtypes` (as is documented that this function will change when new nullable dtypes get added).
> we should change the api but that's a future issue, do we have one?

No, but I will open one.

> is there a test for convert_floating=True & convert_integer=True and it converts floats to ints?

Yes, all possible combo's of all parameters are tested, and we have cases of both integer-like floats as well as actual floats that can't be casted to int
thanks @jorisvandenbossche
CLN: fix flake8 C408 part 2
State: closed

followup of #38078

Labels:
name: Code Style
description: Code style, linting, code_checks
name: Clean

Comments:
Thanks for the PR - do we have an issue for this?
> Thanks for the PR - do we have an issue for this?

I don't think so. Shall I open one?
this is fine, thanks @fangchenli
BUG: name retention in Index.intersection
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

its going to take a few passes to get the behavior consistent across all subclasses.  once done, we can de-duplicate the boilerplate.

Labels:
name: Bug
name: MultiIndex
name: Index
description: Related to the Index class or subclasses
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
pls add a whatsnew as well
whatsnew added + greenish (the same test_area_lim)
DOC: Add behavior for Index argument in DataFrame.loc
State: closed

- [x] closes #36850
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:
name: Docs
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
thanks @hongshaoyang
BUG in Series.interpolate: limit_area/limit_direction kwargs with method="pad"/"bfill" have no effect
State: closed

- [x] closes #26796
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Picking up #31048

Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
hmm, precommit is failing.
@simonjayhawkins i expect to merge this for 1.2 but not blocking rc
thanks @arw2019
REF: implement _should_compare
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Index._is_comparable_dtype is limited because it misses a few cases: object dtype, and sub-dtypes within categorical.  This implements Indes._should_compare, which handles those correctly.  It then implements `_get_indexer_non_comparable` for cases in which we short-circuit non-comparable dtypes.

As a proof of concept, this then uses _should_compare and _get_indexer_non_comparable for `PeriodIndex.get_indexer`.

The behavior change, which this tests, is one that IIUC is a bug.  That is, when we do get_indexer with a method and non-comparable dtypes, we should raise instead of return all minus-ones.

If implemented, we'll be able to use _should_compare to simplify all of the get_indexer, get_indexer_non_unique, and set op methods.

Labels:
name: Index
description: Related to the Index class or subclasses

Comments:
> i don't think this method should raise.

> this is a pretty big change from status quo

I stumbled onto this in part because of this test:

```
def test_reindex_datetimeindexes_tz_naive_and_aware():
    # GH 8306
    idx = date_range("20131101", tz="America/Chicago", periods=7)
    newidx = date_range("20131103", periods=10, freq="H")
    s = Series(range(7), index=idx)
    msg = "Cannot compare tz-naive and tz-aware timestamps"
    with pytest.raises(TypeError, match=msg):
        s.reindex(newidx, method="ffill")
```

and i figured that the way we treat periods with mismatched freq is almost identical to how we treat tzawareness-compat.  Or is dt64tz special in this case?

I'd be OK with not-raising in _get_indexer_non_comparable, just want to get this behavior consistent, bc ATM we're all over the place.
xref #36320
So I'm increasingly convinced that the current PeriodIndex.get_indexer behavior with method != None is wrong:

```
dti = pd.date_range("2016-01-01", periods=3)
pi = dti.to_period("D")
pi2 = dti.to_period("W")

ser = pd.Series(range(3), pi)

>>> ser.reindex(pi2, method="ffill")
2015-12-28/2016-01-03   NaN
2015-12-28/2016-01-03   NaN
2015-12-28/2016-01-03   NaN
Freq: W-SUN, dtype: float64
```

But pi vs pi2 are uncomparable just like pi vs dti, so we should raise the same way:

```
>>> ser.reindex(dti, method="ffill")
[...]
  File "pandas/_libs/algos.pyx", line 450, in pandas._libs.algos.pad
    if nleft == 0 or nright == 0 or new[nright - 1] < old[0]:
TypeError: '<' not supported between instances of 'Timestamp' and 'Period'
```
alright!  get ready to see a bunch of de-duplication and fastpaths coming in.
hit me!
PERF: Index.searchsorted
State: closed

- [x] closes #38083
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Performance
description: Memory or execution speed performance
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Thanks!
REF: use _validate_fill_value in Index.insert
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Index
description: Related to the Index class or subclasses

Comments:
I assume that you are going to move some of this logic to the array/_mixins (or that is the goal)
> I assume that you are going to move some of this logic to the array/_mixins (or that is the goal)

For all our existing ExtensionIndex subclasses the logic is already there.

I haven't 100% decided where I expect this logic to end up, am still tracking down all the places where we do something similar and figuring out if we can align all the behavior. 
PR itself looks ok to me. @jorisvandenbossche @TomAugspurger on the pd.NA comments above (e.g. checking for NA actually will work for datetimelikes).
any more thoughts here?  trying to unify all our casting code
> if the intent is to consoliate this fill logic elsewhere

very much so
BUG: Series.at returning Series with one element instead of scalar
State: closed

- [x] closes #38053
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

I think if we are at this place and len(values) is 1, we want to return a scalar instead of a Series. This happens only because loc is a slice (0,1) with one MultiIndex level instead of a scalar.

Should I create a file to move all MultiIndex at tests to the MultiIndex folder?

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
hmm, lots of things are failing, can you merge master and ping on green.
Good test, caught a bug. Adjusted the code.
@jreback green now
thanks @phofl
BUG: fix wrong error message in deprecated 2D indexing of Series with datetime values
State: closed

Closes #35527

For most Series types, this was properly raising the deprecation warning about 2D being deprecated, but for Series with datetimetz values, this started raising an AssertionError instead of only raising the warning.



Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Deprecate
description: Functionality to remove in pandas

Comments:
LGTM pending green.  I'm OK with this solution, or with working the check into the existing check_ndim.
@jorisvandenbossche if you can update
@jorisvandenbossche if you can update for the small comments above
thanks @jorisvandenbossche 
@meeseeksdev backport 1.1.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 03771a2421b8b7c05983a90bf88f26ce71b1649d
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #38099: BUG: fix wrong error message in deprecated 2D indexing of Series with datetime values'
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-38099-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #38099 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@simonjayhawkins will do the backport
Thanks @jorisvandenbossche
API: CategoricalIndex.append fallback to concat_compat
State: closed

- [x] closes #14586
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

The main thing being changed here is `CategoricalIndex._concat` to better match the behavior of all the other `FooIndex._concat` methods.  Everything else being changed here is just cleanup that this makes possible.

cc @jorisvandenbossche 

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Categorical
description: Categorical Data Type

Comments:
whatsnew added + greenish (test_area_lim)
TST: Suppress http logs in tests
State: closed

- [x] closes #38090
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Confirming that this does the trick.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
still there: https://travis-ci.org/github/pandas-dev/pandas/jobs/746140093
thanks @mroeschke
CLN: testing window namespace
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

It appears we are heading this general direction in pandas imports in tests.

Labels:
name: Code Style
description: Code style, linting, code_checks
name: Window
description: rolling, ewma, expanding

Comments:
yep, though not yet completely enforced. merge on green.
thanks @mroeschke
BUG: merge_ordered fails with list-like left_by or right_by
State: closed

- [x] closes #35269
- [x] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Hello @GYHHAHA! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-11-29 09:11:53 UTC
There also exists two parts should be fixed.
```python
>>>l = pd.DataFrame([['g', 'h', 1], ['g', 'h', 3]], columns=list('GHT'))
>>>l
   G  H  T
0  g  h  1
1  g  h  3
>>>r = pd.DataFrame([[2, 1]], columns=list('TE'))
>>>r
   T  E
0  2  1
```
First, the following unexpected left join is related to wrongly `groupby` .
```python
>>>pd.merge_ordered(l, r, on='T', left_by=['G'])
```
|    | G   | H   |   T |   E |
|---:|:----|:----|----:|----:|
|  0 | g   | h   |   1 | nan |
|  1 | g   | h   |   3 | nan |

Second, unexpected result appears when unseen label exists in `left_by` . (also caused by `try ... groupby` part)
```python
>>>pd.merge_ordered(l, r, on='T', left_by=['G', 'h'])
```
|    | G   | H   |   T |   E | h   |
|---:|:----|:----|----:|----:|:----|
|  0 | G   | h   |   1 | nan | G   |
|  1 | G   | nan |   2 |   1 | G   |
|  2 | h   | nan |   2 |   1 | h   |
|  3 | h   | h   |   3 | nan | h   |

Since they are different bugs from the current list-like issue, thus I will make another PR for the above-mentioned two after this merged.

failures from `test_arithmetic`, unrelated? @jreback 
thanks @GYHHAHA very nice!