BrandonFors/pandas-github-issues huggingface dataset:
API: implement __array_function__ for ExtensionArray
State: closed

xref #26380

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
while adding dispatch from numpy to concat_compat, it was discovered that the axis argument is ignored for EAs. changing this fixes another bug, so the relevant change has been broken off into a separate PR #35038 to be reviewed first and reduce the diff here to just the changes required to get tests passing with \_\_array_function__

on master
```
>>> arr = pd.array([1, 2, 3, None], dtype="Int64")
>>> arr
<IntegerArray>
[1, 2, 3, <NA>]
Length: 4, dtype: Int64
>>>
>>> from pandas.core.dtypes.concat import concat_compat
>>>
>>> concat_compat([arr, arr])
<IntegerArray>
[1, 2, 3, <NA>, 1, 2, 3, <NA>]
Length: 8, dtype: Int64
>>>
>>> concat_compat([arr, arr], axis=1)
<IntegerArray>
[1, 2, 3, <NA>, 1, 2, 3, <NA>]
Length: 8, dtype: Int64
>>>
>>> np.concatenate([arr, arr])
array([1, 2, 3, <NA>, 1, 2, 3, <NA>], dtype=object)
>>>
>>> np.concatenate([arr, arr], axis=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<__array_function__ internals>", line 5, in concatenate
numpy.AxisError: axis 1 is out of bounds for array of dimension 1
>>>
```
Thanks. I think this should probably be a mixin that arrays can opt into by inheriting from it.
for out internal EAs we could opt-in for all. It maybe that we don't need to opt-in for PandasArray and just opt-in for StringArray
> It maybe that we don't need to opt-in for PandasArray and just opt-in for StringArray

opt-in for a subclass of a class that doesn't opt-in seems more trouble than it's worth since would need to override some of the inherited methods. 

so will add tests back for PandasArray and add ArrayFunctionMixin shortly.
tests failing

```

        msg = "axis 1 is out of bounds for array of dimension 1"
        with pytest.raises(np.AxisError, match=msg):
>           np.concatenate([data] * 3, axis=1)
E           Failed: DID NOT RAISE <class 'numpy.AxisError'>
```

the changes to make this work was broken off in #35038 but reverted in #36115

so I either need to add this changes back in here or delete this test.
> Thanks. I think this should probably be a mixin that arrays can opt into by inheriting from it.

after some more thought I not so sure this is a good idea, otherwise we won't be able to use the numpy api in the codebase in the future to avoid special casing EAs in the code which was my motivation for investigating this.

of course, this would be an additional burden on 3rd party EA authors so needs further consideration, xref #32586
On traces of #33982
State: closed

To pay attention on https://github.com/pandas-dev/pandas/pull/33982#issuecomment-643246205. Sorry, I don't know another way now.

/cc @mproszewska , @jreback

Thanks.

Labels:

Comments:
Can you clarify what this is for? I didn't understand from the comment linked
@WillAyd , #33982 introduces `index_col` argument in `_is_potential_multi_index`:
```diff
- def _is_potential_multi_index(columns):
+ def _is_potential_multi_index(
+     columns, index_col: Optional[Union[bool, Sequence[int]]] = None
+ ):
```

There are two usages of this function in Pandas:
1. https://github.com/pandas-dev/pandas/blob/v1.0.5/pandas/io/parsers.py#L1526
2. https://github.com/pandas-dev/pandas/blob/v1.0.5/pandas/io/parsers.py#L1547

#33982 modifies first call only. In that comment I ask if it is correct.
@kuraga - might be worth adding a test to highlight what functionality this changes?
Closing as its not clear what this is or solves; if you can clarify further and add a test case can certainly reopen
@WillAyd , it solves nothing. It demonstrates https://github.com/pandas-dev/pandas/pull/33982#issuecomment-643246205
@kuraga ia this actually a bug?

iow can u hit the line of code u added?
DOC: correction to "size" in the plotting.bootstrap_plot docstring
State: closed

The documentation for plotting.bootstrap_plot says that the size argument must be greater than or equal to the length of the series. Bootstrapping typically uses subsamples of a size less than or equal to the size of the data set, so I corrected the docs accordingly. Note that making size greater than the length of the series throws the following ValueError: Sample larger than population or is negative

Labels:
name: Docs

Comments:
Great thanks @ericgrosz
TST: add test to ensure that df.groupby() returns the missing categories when grouping on 2 pd.Categoricals
State: closed

- [x] closes #23865
- [x] closes #27075 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

1) _New Tests Addeded_
#23865 #27075 both reported that when df.groupby was called and `by` was set to more than one pd.Categorical column, that any missing categories were not returned, even when observed=False. This issue was fixed in #29690. This Pull Request adds tests to make sure that this correct behaviour is enforced by tests. 

2) _New Bug Found_
Testing did reveal one further issue: DataFrameGroupBy.count() returns NaN for missing Categories, when it should return a count of 0.  SeriesGroupBy.count() does return 0, which is the expected behaviour. I have raised an issue for this bug (#35028 ) and marked the test with an xfail. When the bug is fixed, the xfail will cause the tests to fail, and the xfail can be removed. 

3) _Existing Test Changed as it had the Wrong Expected Result_
A similar issue was reported for .sum() in #31422: missing categories return a sum of NaN when they should return a sum of 0. There was a mistake on the existing test for SeriesGroupBy.sum(), as it said the expected output was NaN (see below) when it should have been 0.
https://github.com/pandas-dev/pandas/blob/0159cba6eb14983ab7eaf38ff138c3c397a6fe3b/pandas/tests/groupby/test_categorical.py#L1315

I have changed this so that the expected output is 0 (this is inline with the comment here: https://github.com/pandas-dev/pandas/issues/31422#issuecomment-579826193 ) and marked the tests for .sum() with xfail. When the bug is addressed, the xfail will cause the tests to fail, and the xfails can be removed. 

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Categorical
description: Categorical Data Type

Comments:
@jreback Can I get a another review/approval on this? I'd like to close it off and continue addressing the related bugs. 

Addressed all your previous comments and merged in the latest master with all checks passing. 
thanks @smithto1 very nice, very happy to have follows to fix things :->
CI: add validate_unwanted_patterns to known_third_parties
State: closed

- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

By using `seed-isort-config`, the `known_third_party` setting is automatically populated and alphabetically sorted. This is a cleaner solution which doesn't require manually adding third parties to `setup.cfg`.

Furthermore, it catches an existing inconsistency: currently, `validate_docstrings` is set as a known third party, while `validate_unwanted_patterns` isn't. Neither should be, IMO.

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
We had this and removed it in https://github.com/pandas-dev/pandas/pull/28272 (see some discussions there). I think I struggled the most with it, dunno if it hit other contributors.
@TomAugspurger thanks for pointing me towards that - OK, am happy to not add `seed-isort-config` if it causes issues.

Should `validate_docstrings` be removed from the list of known_third_parties though?
Don't know. It's not in the pandas namespace, so probably not? Don't have a strong preference though.
Sure - have removed seed-isort-config and added `validate_unwanted_patterns` to the list of known 3rd parties (I presume they should either both be there, or neither?)
I think this is fine
Thanks!
BUG-Fix: AssertionError when slicing MultiIndex and setting value of …
State: closed

- [x] closes #34870 
- [x] tests added / passed (1 / 1)
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex
name: Stale

Comments:
@dsaxton, please, can you watch it. I looked in more detail. In variables, `start_slice` and `end_slice` are lists with coordinates. Therefore, we get an error when checking for a scalar. Having studied the code, I did not find the reasons why we cannot process them.
> @dsaxton, please, can you watch it. I looked in more detail. In variables, `start_slice` and `end_slice` are lists with coordinates. Therefore, we get an error when checking for a scalar. Having studied the code, I did not find the reasons why we cannot process them.

I'm not very familiar with this part of the code base so take my advice with a grain of salt, but as far as I can tell start_slice and end_slice _should_ be scalars so we don't want to just remove these assertions as you're doing here.

I would instead look upstream of this to figure out why this is happening (it does seem that the Index is mangled by the time we get here and is no longer a MultiIndex, if that's any clue).
@dsaxton, I rolled back the changes and took your advice into account. You can view?
@LuckyDenis can you address comments and add a release note
@LuckyDenis can you address @jreback comment https://github.com/pandas-dev/pandas/pull/35018#discussion_r468680783
Closing as stale
CLN: GH29547 format with f-strings
State: closed

xref #29547,#34914
replace .format() for f-strings in the following:

pandas/tests/series/indexing/test_numeric.py

Labels:

Comments:
Thanks but I think the use of format here is better than a lambda. Basically any time we have to defer string evaluation like this format is better
Thanks @WillAyd .

i have a question though-when i made the recent pull request, the two CI were green but all others were red.
when i tried to follow the error i couldn't understand what the errors were.
perhaps you can tell me. what does it mean when there is a situation like that? how can i understand that better?
If you click on the Checks tab of the PR you should be able to see all of the failed builds. Click through them and you should see the test that failed
Fix issue #35010: Double requirement given for fsspec
State: closed

- [ ] closes #35010 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Build
description: Library building on various platforms
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
thanks, ping on green.
thanks @SanthoshBala18
Fix Issue #34923: Inferred dtype at the end of df explode method
State: closed

- [ ] closes #34923 
- [ ] tests added / passed
          pandas/tests/frame/methods/test_explode.py, method test_inferred_dtype
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Needs Discussion
description: Requires discussion from core team before further action

Comments:
Hello @SanthoshBala18! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-07-20 11:26:43 UTC
you may need to update the example in the docs / doc-string as well
@jreback, merged the master branch..
 I'm -1 on this. This is a breaking change and not a bug.

the change for lists of ints with some rows containing empty lists now being cast to float should wait till a major release and maybe use the nullable integer type instead.

current
```
>>> pd.__version__
'1.1.0rc0'
>>>
>>> s = pd.Series([[1, 2, 3, 4], [], [5, 6, 7]])
>>> s
0    [1, 2, 3, 4]
1              []
2       [5, 6, 7]
dtype: object
>>>
>>> res = s.explode()
>>> res
0      1
0      2
0      3
0      4
1    NaN
2      5
2      6
2      7
dtype: object
>>>
>>> type(res.iloc[1])
<class 'int'>
>>>
```

this PR
```
>>> pd.__version__
'1.1.0rc0+13.gf6144512d'
>>>
>>> s = pd.Series([[1, 2, 3, 4], [], [5, 6, 7]])
>>> s
0    [1, 2, 3, 4]
1              []
2       [5, 6, 7]
dtype: object
>>>
>>> res = s.explode()
>>> res
0    1.0
0    2.0
0    3.0
0    4.0
1    NaN
2    5.0
2    6.0
2    7.0
dtype: float64
>>>
>>> type(res.iloc[1])
<class 'numpy.float64'>
>>>

```
closing this. see my comments on the issue (basically need to add an option `downcast=` instead here.
CLN: remove redundant code in IndexOpsMixin.item
State: closed

the code removed in this PR was added (with tests) in #30175. 

in a latter PR, #31506, IndexOpsMixin.\_\_iter__  was modified, making the special case in IndexOpsMixin.item redundant.

cc @jbrockmendel 

Labels:
name: Clean

Comments:
thanks @simonjayhawkins
BUG: DataFrame.melt gives unexpected result when column "value" already exists
State: closed

- [ x] closes #34731 
- [x ] tests added / passed
- [ X] passes `black pandas`
- [ x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ x] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Deprecate
description: Functionality to remove in pandas

Comments:
> Linting .py code
> 16
> ##[error]./pandas/core/apply.py:9:1:F401:'pandas._libs.reduction as libreduction' imported but unused
> 

should I remove this line as part of this PR? "libreduction" isn't mentioned anywhere else in the file pandas/core/apply.py , but on the other hand, its out of scope

Can you merge master to fix CI issue?
thanks @pizzathief 
@pizzathief want to make a PR to enforce this deprecation?
CLN: remove libreduction.Reducer
State: closed


Labels:
name: Performance
description: Memory or execution speed performance
name: Clean

Comments:
nice, merge master (as previously cleared a lint issue)
Any performance diff in doing this? 
> Any performance diff in doing this?

No.  We removed the only non-test usages of this in #34913 and #34909, which were both measured as perf-neutral.
BUG: item_cache not cleared on DataFrame.values
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry



Labels:
name: Performance
description: Memory or execution speed performance
name: Clean

Comments:
any perf impact? (i would actually expect some small regressions)
```
In [1]: import pandas as pd                                                                                                                                                                                        


In [2]:                                                                                                                                                                                                            

In [2]: df = pd.DataFrame([1, 2])                                                                                                                                                                                  

In [3]: %timeit df.values
2.25 µs ± 42.9 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- master
3.82 µs ± 106 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- PR
```
hmm, do we have any to_numpy() benchmarks? just to see not blocking this PR
> do we have any to_numpy() benchmarks

Looks like we do not
> > do we have any to_numpy() benchmarks
> 
> Looks like we do not

kk happy to merge and can create issue for benchmarks (as a good first - people like to do those)
issue opened
thanks
@jbrockmendel caused a 100x slowdown in replace.Convert.time_replace (https://pandas.pydata.org/speed/pandas/index.html#replace.Convert.time_replace?p-constructor=%27Series%27&p-replace_data=%27Timestamp%27&commits=dbfbef7eaea7f23a42d9a65344b5745fc055f5b4). Is that expected?
>  caused a 100x slowdown in replace.Convert.time_replace  [...] Is that expected?

I'd expect something closer to the 70% slowdown we see in `df.values`.  100x is surprising.
Opened https://github.com/pandas-dev/pandas/issues/35053 to track this.
Hey I am new to open source. Can I take up this issue?

@B417037 this is a merged PR, not an issue to take up. Have a look at https://github.com/pandas-dev/pandas/labels/good%20first%20issue
TST: df.loc[:, 'col'] returning a view, but df.loc[df.index, 'col'] returning a copy
State: closed

- [x] closes #15631
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
I'm not sure about this - are we expecting that this loc call copies? @mroeschke @jreback 
@arw2019 are you still working on this? 
@erfannariman yes I think this is ready to go, waiting for review
@WillAyd 

> I'm not sure about this - are we expecting that this loc call copies? @mroeschke @jreback 

yes this has been the case for a long time
thanks @arw2019
DOC/TST: DataFrame constructor with a list of DataFrames
State: closed

Closes #32289 by removing the failing tests and asserting that we raise For NumPy>=1.19. See https://github.com/pandas-dev/pandas/issues/32289#issuecomment-649492024 for details, but the tldr is that DataFrames are now treated identically to 2D ndarrays. 

One thing I wasn't sure of: where to document this. IMO this is too minor of an edge case to warrant anything in the docstring / user guide.

Labels:

Comments:
Merging this in a few hours since this is causing our wheel builds to fail.
TYP: remove inappropriate use of cast
State: closed

the cast was silencing `error: Argument 2 to "zip" has incompatible type "Union[Sequence[Union[str, int]], Mapping[Optional[Hashable], Union[str, int]]]"; expected 
"Iterable[Union[str, int]]"`

This error is because col_space is defined as `Union[str, int, Sequence[Union[str, int]], Mapping[Label, Union[str, int]]]` and the `elif isinstance(col_space, dict):` would not catch non-dict mappings, which would go though the else.

so the else is non-dict mappings and sequences and the cast to sequence is not safe.

```
>>> df = pd.DataFrame(np.random.random(size=(3, 3)), columns=["a", "b", "c"])
>>>
>>> print(df.to_string(col_space={"b": 20}))
          a                    b         c
0  0.382920             0.057121  0.862742
1  0.579339             0.391014  0.907678
2  0.340584             0.889387  0.922690
>>>
>>> from collections import UserDict
>>>
>>> print(df.to_string(col_space=UserDict(b=20)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\simon\pandas\pandas\core\frame.py", line 838, in to_string
    formatter = fmt.DataFrameFormatter(
  File "C:\Users\simon\pandas\pandas\io\formats\format.py", line 601, in __init__
    raise ValueError(
ValueError: Col_space length(1) should match DataFrame number of columns(3)
>>>
```


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @simonjayhawkins
BUG: Cannot create third-party ExtensionArrays for datetime types (xfail)
State: closed

- [x] closes #34986
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Stale

Comments:
This is just the failing test for now, happy to implement a fix if someone could tell me the location where this should be fixed.
@jbrockmendel 
xfail added, CI is now happy.
can you merge master and we'll see if we can get this in
@jbrockmendel Rebased and all green except one Windows job that timeouted.
I think the edit to get_block_type in #34683 might fix the test that fails here.  can you confirm?  if that is fixed, presumably the rest of the EA test suite still needs to be enabled for this EA?
> I think the edit to get_block_type in #34683 might fix the test that fails here. can you confirm? if that is fixed, presumably the rest of the EA test suite still needs to be enabled for this EA?

Yes, merging in #34683 fixes the test. 

I'm not sure whether it would be really worth to get the full suite running for this test EA. It is basically here to check for the regression but getting the whole suite to pass would be a lot more work that I don't see worthwhile currently.
> I'm not sure whether it would be really worth to get the full suite running for this test EA. It is basically here to check for the regression but getting the whole suite to pass would be a lot more work that I don't see worthwhile currently.

totally reasonable.  i guess we can merge this now and then if/when #34683 makes this pass we can revisit getting other bits working.

cc @jreback
@xhochy can you merge master, hopefully we'll get the CI green and can get this in
Hello @xhochy! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-01-14 13:50:47 UTC
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@jbrockmendel @jreback Rebased and removed xfail as it is working now.
thanks @xhochy
ENH: concat of nullable int + bool preserves int dtype
State: closed

Closes #34095

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks. Does the original whatsnew from the other PR describe this change as well?
> Thanks. Does the original whatsnew from the other PR describe this change as well?

I think so yes, it's still in general about preserving EA dtype in concat. But I added the issue number to the exsting whatsnew.
Merged master.

@jreback was the discussion in https://github.com/pandas-dev/pandas/pull/34985#discussion_r449643794 resolved?
yes this is fine to merge