BrandonFors/pandas-github-issues huggingface dataset:
Revert "REF: back IntervalArray by a single ndarray (#37047)"
State: closed

This reverts commit 9cb372376fc78aa66e2559de919592007b74cfaa.

See discussion in https://github.com/pandas-dev/pandas/pull/37047/

@jbrockmendel can you give this a check? (there were already several conflicts)

Labels:
name: Refactor
description: Internal refactoring of code
name: Interval
description: Interval data type

Comments:
I'll take a look, tentatively looks OK.

By funny coincidence, I _just_ got the first of the promised perf improvements working.
@jbrockmendel ok here?
looks fine
>  I just got the first of the promised perf improvements working.

Do you have a branch for it that you can push? I am curious to see whether the principles of the speed-up also couldn't be applied to the "2 1D-arrays" model
> Do you have a branch for it that you can push? I am curious to see whether the principles of the speed-up also couldn't be applied to the "2 1D-arrays" model

shortly, i hope.  The approach is to do `self._combined.view("complex128")` and then do set operations on that to avoid casting to object.
I can imagine that an additional copy to create the combined array could still be worth it compared to casting to object
ENH: support 2D in DatetimeArray._from_sequence
State: closed

Broken off from a branch that fixes #37682, the constraint on which is that _validate_setitem_value needs to handle 2D.

Labels:
name: Enhancement
name: Refactor
description: Internal refactoring of code

Comments:
Thank you for weighing in Joris.  As I said in the OP, this is part of fixing #37682.
hmm, yeah i think i agree with @jorisvandenbossche here, we should either go all in on 2D or not. can we just process column by column instead?
> can we just process column by column instead?

we _can_, but it means putting more logic in Blocks which is the opposite of what im trying to do
looking at this again it is fine. we do this in all kinds of places where we can potentially have 1d or 2d inputs.

unless objecting by @jorisvandenbossche and i think would need a concrete proposal that acheives the same aim, will merge tomorrow.
this is fine
Using os.PathLike instead of pathlib.Path (#37979)
State: closed

- [x] closes #37979 
- [x] tests passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Replaces `pathlib.Path` with `os.PathLike` to bring behaviour further in-line with the [documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html)

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@Abhi-H see if you can make mypy happy.
Working on fixing the mypy errors.
@arw2019 Is there still a merge error? I don't quite understand what those are, so a link or guidance on how I could resolve it would be very helpful. I am assuming that this is related to the git side of things as the code seems to pass all checks, so I can quickly resolve it!
Also, just checked and the changes I was making are there. 
With some git-magic and help from a friend, I think I fixed the merge conflict. Let me know if this is fine!
I am a little stuck here. The tests that are failing in these builds are not failing when I run them locally. What could be causing this?
I think the only test that is failing in this PR but not on master is `test_to_csv_compression_encoding_gcs`. Can you run that test locally to see whether it works for you `pytest -k test_to_csv_compression_encoding_gcs pandas/tests/io/test_gcs.py`. I will test it later today on my machine on linux.

edit: I don't get the above error. The removed `hasattr(...)` shouldn't have an impact as the testcase uses already a string. Maybe just rebase and hope for the best: `git pull --rebase upstream master`
Thanks for checking it out, I tried and didn't get the error either when running locally either. Have rebased and pushed a second time.
thanks @Abhi-H very nice!
BUG: Fixes plotting with nullable integers (#32073)
State: closed

- [x] closes #32073
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry




Labels:
name: Visualization
description: plotting
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
This is the first time I am contributing to pandas open source project. Please let me know if I can improve anything on this commit. 
@jreback the branch has the latest code and all the checks passed. 
thanks @rkc007
REF: Implement isin on DTA instead of DTI
State: closed


Labels:
name: Refactor
description: Internal refactoring of code
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
> Why add it to the array class while for none of our other EAs it is implemented on the array?

1) it is implemented on Categorical
2) jreback requested the move in #37528
3) i intend to de-duplicate code in `isin`, `equals`, and `_validate_listlike`
OK, sounds good
thanks @jbrockmendel 

yeah I am +1 on adding .isin too the base class (and coercion would go thru `_from_sequence` so likely this would mostly be a base-class method (not much need to override)
BUG: loc returning wrong elements for non-monotonic DatetimeIndex
State: closed

- [x] closes #33146
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

That one was tricky. When only one existing key was given or both string parts exist, ``Index.slice_indexer`` is not raising KeyError but selecting a lof of erroneous values

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
have to look at this in detail
Hello @phofl! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-01-03 21:51:10 UTC
thanks @phofl very nice.
ENH: str extract with default value
State: closed

- [x] closes #38001 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: Strings
description: String extension data type and string data
name: Stale

Comments:
Hello @erfannariman! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-05-10 09:42:30 UTC
@jreback any chance we can decide what we want to do with this PR? If you think this change is not needed, It's fine for me to close it.
i'd really like a real-world usecase of this. The work-aroudn is actually very simpler (even simpler that what you posted), so not convinced this all that useful. that said prob doesnt' hurt.

@erfannariman yeah i guess this is ok, can't hurt. rename default -> fill_value (as that is the term we use else where) and move the note to 1.3. 
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@erfannariman closing this, if you think of a real case for this, happy to reopen. But agree with Jeff that even if this makes sense, adding more complexity to pandas for a reasonable use case that no user may ever face, it's not worth
> @erfannariman closing this, if you think of a real case for this, happy to reopen. But agree with Jeff that even if this makes sense, adding more complexity to pandas for a reasonable use case that no user may ever face, it's not worth

technically it's probably straightforward, we already pass na in other methods and handle it in the wrap_result and _str_map methods (as na_value)

i'm not sure a real world case is required to justify this as accepting a na parameter would make str.extract consistent with other str accessor methods. 
I'll reopen this, but I think this enhancement should wait until after str.extract is dispatched to array and the _wrap_result and str_map methods are re-used. #41372
@erfannariman can you resolve conflicts and can take another look
@erfannariman closing as stale. ping if you want to pick this up again.
DOC: Deprecate null_counts parameter of DataFrame.info
State: closed

- [x] closes #36805
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

### Background. 

> Might be worth it actually - I _think_ people are only likely to use `.info` in an interactive session, rather than as part of production code, so perhaps it is worth going through the deprecation process to make this argument clearer

* xref https://github.com/pandas-dev/pandas/issues/36805#issuecomment-702754290
* Deprecate `null_counts` parameter of DataFrame.info in favor of `show_counts` and add warning.
* Add whatsnew and add tests to test for FutureWarning. Move usages of null_counts in tests to show_counts.

Doc preview: 

<img width="755" alt="Screenshot 2020-11-22 at 6 19 05 PM" src="https://user-images.githubusercontent.com/19281800/99901143-63512400-2cef-11eb-846a-447050de6d8f.png">


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @arw2019 , I added tests and a whatsnew entry. 
looks fine, @jorisvandenbossche ok here?
just merged some changes that will affect this PR, can you merge master and update: https://github.com/pandas-dev/pandas/pull/37868
thanks @hongshaoyang very nice
@hongshaoyang want to make a PR enforcing this deprecation?
BUG: crosstab with duplicate column or index labels
State: closed

- [x] closes #22529
- [x] tests added / passed
- [x] passes black pandas
- [x] passes git diff upstream/master -u -- "*.py" | flake8 --diff
- [x] whatsnew entry

Picking up from #28474 

cc @jreback in case this can go in in time for 1.2

Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
I tried consolidating the tests using `eval` but for a reason I can't atm figure out this doesn't work:
``` python
    @pytest.mark.parametrize("args_res, args_exp, axis", (
        ["s3, [s1, s2_foo]", "s3, [s1, s2_bar]", 1],
        ["[s1, s2_foo], s3", "[s1, s2_bar], s3", 0],
    ))
    def test_crosstab_duplicate_labels(self, args_res, args_exp, axis):
        # GH 22529

        s1 = Series(range(3), name="foo")
        s2_foo = Series(range(1, 4), name="foo")
        s2_bar = Series(range(1, 4), name="bar")
        s3 = Series(range(3), name="waldo")

        result = crosstab(*eval(args_res))

        expected = crosstab(*eval(args_exp))
        expected = expected.rename_axis(columns={"bar": "foo"}, axis=axis)

        tm.assert_frame_equal(result, expected)

```

Update: it's because in rename_axis we pass alternately `columns` and `index` kwargs. 
Green + addressed comments
thanks @arw2019 and @cuchoi
[WIP] DOC: MultiIndex EX01 errors
State: closed

- [x] xref #37875, #27977
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs
name: MultiIndex

Comments:
Hello @xinrong-databricks! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-11-22 22:07:43 UTC
thanks @xinrong-databricks keep em coming!
Thank you @jreback for review and merging!
Bug in DataFrame.loc returning elements in wrong order when indexer is differently ordered than object
State: closed

- [x] closes #34603
- [x] closes #31330
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
LGTM thanks for hunting these down @phofl 
needs rebase otherwise LGTM
Thx for rebasing
thanks @phofl
TST: Series construction from ExtensionDtype scalar
State: closed

- [x] closes #28401
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
this looks fine.  should there be an analogous test in tests.extension?
> this looks fine. should there be an analogous test in tests.extension?

Hmmm do you mean something like this?
``` python
In [7]: pd.array(pd.Interval(0, 1))
(...)
ValueError: Cannot pass scalar '(0, 1]' to 'pandas.array'.
```
We test this here:
https://github.com/pandas-dev/pandas/blob/52a1725965cb469de618ad56af467dc0aba53a87/pandas/tests/arrays/test_array.py#L287-L290
and 
https://github.com/pandas-dev/pandas/blob/52a1725965cb469de618ad56af467dc0aba53a87/pandas/tests/arrays/floating/test_construction.py#L85-L108

thanks @arw2019 

FYI there maybe other places where we can use this fixture, if its simple, would take a PR, if not, pls open an issue.
> FYI there maybe other places where we can use this fixture, if its simple, would take a PR, if not, pls open an issue.

Working on it. Probably will be a single PR but will see
TST: add test to verify column does not lose categorical type when using loc
State: closed

- [x] closes #16360
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry (not applicable)

Added a test to verify the behavior described in issue #16360. However, I'm not 100% sure about the location of the test.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Categorical
description: Categorical Data Type

Comments:
failed test is unrelated to my changes - do you collect flaky tests somewhere? i.e. `pandas/tests/plotting/frame/test_frame.py::TestDataFramePlots::test_area_lim`
thanks @fgebhart very nice
CLN/TST: delegate StringArray.fillna() to parent class + add tests
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Strings
description: String extension data type and string data

Comments:
Does this need a whatsnew?
> Does this need a whatsnew?

I don't think it is necessarily needed, it's only a change in the error message I think?
I pared down this PR to just the fillna test for strings. Discussion on `_validate_setitem_value` is housed in #36226 (happy to work on that in a follow-on unless somebody else wants to)
thanks @arw2019
REGR: fix inplace operations for EAs with non-EA arg
State: closed

- [x] closes #37910 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
@arw2019 can you add a release note to the regression section of doc\source\whatsnew\v1.1.5.rst
thanks @arw2019 
@meeseeksdev backport 1.1.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 20f7ffa0e9a87fe207a316574d7343673b99d9d9
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #37986: REGR: fix inplace operations for EAs with non-EA arg'
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-37986-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #37986 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BUG: IntervalArray.astype(categorical_dtype) losing ordered
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Categorical
description: Categorical Data Type

Comments:
looks fine, can you add a whatsnew note to document, ping on green.
whatsnew added, greenish (same test_area_lim travis failure)
ping
DEPR: how keyword in PeriodIndex.astype
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Period
description: Period data type
name: Deprecate
description: Functionality to remove in pandas

Comments:
thanks
Deprecate inplace in Categorical.remove_categories
State: closed

- [x] xref #37643
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Categorical
description: Categorical Data Type
name: Deprecate
description: Functionality to remove in pandas
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
ok let's resolve the discussion on the issue first here.
@arw2019 I fixed the 'whatsnew' entry as you asked.

Updated to the latest master and resolved, hopefully all, unit test problems.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
needs rebase, otherwise LGTM
@jbrockmendel 
I have updated this pull request to the up-to-date main branch
one comment, otherwise looks ready
thanks @OlehKSS let's merge on green.
thanks @OlehKSS
BUG: CategoricalIndex.where nulling out non-categories
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

ATM CategoricalIndex.where has some idiosyncratic behavior:

```
ci = CategoricalIndex(["a", "b", "c", "d"])
mask = np.array([True, False, True, False])

>>> ci.where(mask, 2)
CategoricalIndex(['a', nan, 'c', nan], categories=['a', 'b', 'c', 'd'], ordered=False, dtype='category')
```

This makes that call raise instead.

Index.where is only used in one place in reshape.merge.  Might be worth deprecating+privatizing.


Labels:
name: Bug
name: Categorical
description: Categorical Data Type

Comments:
also pls add a whatsnew note
whatsnew added + green
TST: add messages to bare pytest raises in pandas/tests/io/pytables/test_timezones.py
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This PR adds messages to the bare pytest raises in pandas/tests/io/pytables/test_timezones.py. This PR references https://github.com/pandas-dev/pandas/issues/30999.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
> Nice, thanks! looks good to me pending green

Thanks!

> feel free to take on more than one file in the same PR if the changes aren't too big

Will do, I think that will work for a couple of the other files.
@MarcoGorelli thanks for your suggestion, I made the change and now all the checks pass.
thanks @marktgraham
BUG: fix astype conversion string -> float
State: closed

- [x] Is a step towards closing #37626
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Fixes the `string` -> `float` conversion as discussed in #37626 in presence of `pd.NA`. I used the same approach as used in `string` -> `Int64`. Note that differently to the `Int64` case, we return a `numpy.ndarray`, not a `pd.array`.

I added a test that compares `pd.Series`. Replacing this with `pd.array` the `tm.assert_numpy_array_equal` assertion will raise, as `expected` is of type `FloatingArray`. IIUC this is a new feature of pandas 1.2.0. Should we return this in `.astype("float")` by default?

Labels:
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Sure.

> Would you consider fixing the similar issue with astype("int") conversion as well in a separate PR?

Can you expand on the issue and, if not obvious, with the expected behaviour?
> 
> 
> Sure.
> 
> > Would you consider fixing the similar issue with astype("int") conversion as well in a separate PR?
> 
> Can you expand on the issue and, if not obvious, with the expected behaviour?

Similar to float.

```
In [1]: import pandas as pd
In [2]: ser = pd.Series([1, pd.NA, 2], dtype="string")
In [3]: ser.astype("int")
...
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NAType'
```
> In [1]: import pandas as pd
> In [2]: ser = pd.Series([1, pd.NA, 2], dtype="string")
> In [3]: ser.astype("int")
> ...
> TypeError: int() argument must be a string, a bytes-like object or a number, not 'NAType'

Hmm. As far as I understood, `int` (the non-extension array dtype) does not support missings. See:

```
In [1]: import numpy as np
In [2]: import pandas as pd
In [3]: pd.Series([1.0, np.nan], dtype="float").astype("int")
ValueError: Cannot convert non-finite values (NA or inf) to integer
```
However, using the nullable extension array dtype `Int64`, the conversion works:
```
In [4]: pd.Series(["1", pd.NA], dtype="string").astype("Int64")
Out[4]: 
0       1
1    <NA>
dtype: Int64
```

Would you prefer if above (`string` with `pd.NA` to `int`) raised a `ValueError` instead?

**Edit**: Were you possibly referring to`object` -> `Int64` as in `pd.Series(["1", "2"]).astype("Int64")`?


Unrelated: Any idea what's up with the failing tests?

@jreback Could you have another look?
thanks @mlondschien
TST: add nullable array frame constructor dtype tests
State: closed

- [x] closes #28424
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
thanks @arw2019
PERF: IntervalArray.argsort
State: closed

IntervalIndex.argsort has a more performant implementation (at least for default kwargs).  This just moves that up to IntervalArray.

```
In [2]: idx = pd.IntervalIndex.from_breaks(range(10**4))

In [3]: arr = idx._data

In [4]: %timeit arr.argsort()
24.5 ms ± 458 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)     # <-- master
79 µs ± 1.64 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)   # <-- PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Interval
description: Interval data type

Comments:
do we have sufficient asv's for this?
> do we have sufficient asv's for this?

no, but itll be relatively easy to add mroe since there is a SortIntegerArray class that has a time_argsort, we can extend that