BrandonFors/pandas-github-issues huggingface dataset:
BUG: Make ExtensionArray:_can_hold_na a class attribute
State: closed

- [x] closes #20761 
- [x] tests added / passed
  - pandas/tests/extension/decimal/test_nona.py
    - This tests an example where `_can_hold_na==False`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry
  - Not included as it is a fix to the new `ExtensionArray` interface

Supersedes #20801 by removing ops functionality from there.



Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Let's put the discussion here to not hide it in a review comment:

So I questioned what should happen when NAs are introduced (eg reindex) in ExtensionArray that has `_can_hold_na=False`.
I suppose to raise an error? Or coerce to object?

Currently, in pandas we have ints, and they are upcasted to float. We also have a BoolBlock that cannot hold NaNs, so a boolean Series gets converted to object Series when NaNs are introduced:

```
In [49]: pd.Series([True, False])
Out[49]: 
0     True
1    False
dtype: bool

In [50]: pd.Series([True, False]).reindex([1, 2])
Out[50]: 
1    False
2      NaN
dtype: object
```

So this is some prior art. But it would also mean you loose your ExtensionArray/ExtensionDtype information.

So remark: if we currenlty don't have users of `_can_hold_na=False`, we should maybe consider removing the feature for now (meaning this will always be True) until somebody comes up with a good use case.

@Dr-Irv : 

> Well, I've come full circle on this.  In my case, I thought that I would not want NA in my arrays. My objects stored in the arrays have no NA value that makes sense. And then when you brought up `reindex` as an example, I realized that I would use things like `shift` and would want the NA to be filled in, but keeping the `ExtensionArray` class as the holder, meaning that I need `_can_hold_na` as True.  I'd want `ExtensionArray` as holding it so that arithmetic would work, as coercing to an `object` array would mess up the arithmetic. So I'd have no problem if we dropped `_can_hold_na` from the `ExtensionArray` design, and tell people that they must define a _na_value(), which is then used as the "fill in" when we do things like `reindex` and `shift`.  And because there are pandas operations that need some NA value for fill in purposes, requiring it on `ExtensionArray` shouldn't be an issue (IMHO).
>
> If we agree that getting rid of `_can_hold_na` is the thing to do, I'll modify this PR to totally get rid of it (and the associated tests).

cc @TomAugspurger @jreback 

i would not add API unless we have a need for it
so yes would remove until / until is necessary 
Closing due to new PR #20819
CLN: Continued indexes/test_base.py cleanup
State: closed

progress towards #20812

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20813?src=pr&el=h1) Report
> Merging [#20813](https://codecov.io/gh/pandas-dev/pandas/pull/20813?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/8bee97a81764c5211d719d61a62424b1edfacd80?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/20813/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&width=650&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/20813?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #20813   +/-   ##
=======================================
  Coverage   91.77%   91.77%           
=======================================
  Files         153      153           
  Lines       49265    49265           
=======================================
  Hits        45215    45215           
  Misses       4050     4050
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.16% <ø> (ø)` | :arrow_up: |
| #single | `41.86% <ø> (ø)` | :arrow_up: |


------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20813?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20813?src=pr&el=footer). Last update [8bee97a...1acfda1](https://codecov.io/gh/pandas-dev/pandas/pull/20813?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

thanks @WillAyd
Fixed typo
State: closed

I believe it should be either `Then another Python operation ``dfmi_with_one['second']`` happens.` or `Then another Python operation ``dfmi_with_one['second']`` selects the series indexed by ``'second'``.` .
Not `Then another Python operation ``dfmi_with_one['second']`` selects the series indexed by ``'second'`` happens.`.


Labels:
name: Docs

Comments:
thanks!
Index union sort hack
State: closed

- [x] closes #17376
- [x] tests added / passed
- [x] passes `git diff master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This is an attempt to adapt #17378 according to @gfyoung 's suggestions.

(I do plan to investigate more in depth the problem in Windows, I just won't be able to do it soon)

Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
@toobaz : Just to confirm: does this supersede #17378 (i.e. should we close that?)

Curious to see if AppVeyor will be more accommodating to this PR.
> @toobaz : Quick question: does this supersede #17378 (i.e. should we close that?)

... if this passes tests, I guess we will.
> ...if this passes tests, I guess we will.

That's a good enough answer for me. 👍 
closing as stale, if you want to continue working, pls ping and we can re-open. you will need to merge master.
BUG: DataFrame.from_records with empty rec array
State: closed

keep the dtypes

Closes https://github.com/pandas-dev/pandas/issues/20805

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
@WillAyd comments, and a lint issue. otherwise lgtm.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20806?src=pr&el=h1) Report
> Merging [#20806](https://codecov.io/gh/pandas-dev/pandas/pull/20806?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/41db52730b47ff1d707004decd1214fb87e5c533?src=pr&el=desc) will **increase** coverage by `0.02%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/20806/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/20806?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #20806      +/-   ##
==========================================
+ Coverage   91.81%   91.83%   +0.02%     
==========================================
  Files         153      153              
  Lines       49318    49319       +1     
==========================================
+ Hits        45279    45294      +15     
+ Misses       4039     4025      -14
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.23% <100%> (+0.02%)` | :arrow_up: |
| #single | `41.87% <0%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/20806?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/20806/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.17% <100%> (ø)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/20806/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `84.79% <0%> (+0.2%)` | :arrow_up: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/20806/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `66.81% <0%> (+1.73%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20806?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20806?src=pr&el=footer). Last update [41db527...29af436](https://codecov.io/gh/pandas-dev/pandas/pull/20806?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Hmm I can't reproduce the failures locally with those versions of python / NumPy.

May end up pushing this. The `Series([])` dtype change (which this is for) isn't critical for 0.23.
can you rebase
closing as stale. if you'd like to continue, pls ping.
DEPR: Warn on empty Series dtype inference
State: closed

We're changing `Series([])` to be `object` instead of `float`. This does the warning

- [x] xref https://github.com/pandas-dev/pandas/issues/17261 (leave open till we make the change)
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: API Design

Comments:
FYI, d80e1c7dea07303c2aacef32afbd6ccfc6fd47bc has the changes. Everything else is catching / fixing warning internal to pandas. This ended up being a decent-sized change.
Sorry about the empty commits, will fix that up.

Still going through and squashing warnings. There are a few cases internally where we construct an empty series. e.g.

```python
In [1]: import pandas as pd

In [2]: pd.Series([], dtype='object').groupby([]).apply(lambda x: 1)
Out[2]: Series([], dtype: float64)
```

How should we handle that?

1. Leave as float64, specify the dtype to keep compat
2. Leave as float64, no dtype to generate a warning

The problem w/ 2 is that there's nothing the user can do to silence the warning.
@TomAugspurger can you close or update
@jreback @jorisvandenbossche do you think we want to change this before 1.0?
@TomAugspurger can you merge master
There's a decent amount of work to do here to avoid the warnings
internally. Will try to make time.

On Sun, Nov 18, 2018 at 4:07 PM Jeff Reback <notifications@github.com>
wrote:

> @TomAugspurger <https://github.com/TomAugspurger> can you merge master
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/20802#issuecomment-439730129>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIlxE1_nFXT33ZyVrGy5GKVNKLTv7ks5uwdopgaJpZM4Tgluf>
> .
>

@TomAugspurger keep this one open?
Probably not :/ I suppose we can just live with this. A little artifact of the time Series used to be an ndarray.
BUG: Coerce to object for mixed concat
State: closed

- [x] closes #20762
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20799?src=pr&el=h1) Report
> Merging [#20799](https://codecov.io/gh/pandas-dev/pandas/pull/20799?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/31e77b0b4129134d20cd5027dac0adb9fbd2dca0?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/20799/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/20799?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #20799      +/-   ##
==========================================
- Coverage   91.85%   91.85%   -0.01%     
==========================================
  Files         153      153              
  Lines       49310    49322      +12     
==========================================
+ Hits        45292    45303      +11     
- Misses       4018     4019       +1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.24% <100%> (ø)` | :arrow_up: |
| #single | `41.89% <50%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/20799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/dtypes/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/20799/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbmNhdC5weQ==) | `99.18% <100%> (+0.01%)` | :arrow_up: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/20799/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.63% <0%> (-0.05%)` | :arrow_down: |
| [pandas/core/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/pull/20799/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9tdWx0aS5weQ==) | `95.07% <0%> (+0.01%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20799?src=pr&el=footer). Last update [31e77b0...615399c](https://codecov.io/gh/pandas-dev/pandas/pull/20799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Thanks!
EA.values warning test
State: closed

xref #20735 (doesn't close)

Here's a local "test" for it.

```
pytest pandas/tests/extension/test_values.py -k test_no_values
==================================================================== test session starts ====================================================================
platform darwin -- Python 3.7.0b3, pytest-3.5.0, py-1.5.3, pluggy-0.6.0
rootdir: /Users/taugspurger/sandbox/pandas, inifile: setup.cfg
plugins: xdist-1.22.2, forked-0.2, cov-2.5.1
collected 9 items / 8 deselected

pandas/tests/extension/test_values.py F                                                                                                               [100%]

========================================================================= FAILURES ==========================================================================
______________________________________________________ TestBaseInterfaceTests.test_no_values_attribute ______________________________________________________

self = <pandas.tests.extension.test_values.TestBaseInterfaceTests object at 0x112ed4748>, data = JSONArary([{'A': [1, 2], 'B': [3, 4]}])

    def test_no_values_attribute(self, data):
        # GH-20735
        # Currently, pandas has places where we accepts Union[ndarray, EA]
        # but in practice expect an ndarray, since we get `data.values.dtype`.
        # This warns users to avoid using a `.values` attribute that is not
        # an ndarray-like
        if hasattr(data, 'values') and not hasattr(data.values, 'dtype'):
            msg = ("ExtensionArray contains a 'values' attribute that does "
                   "not have a dtype attribute. This may cause issues in "
                   "pandas' internals.")
>           raise ValueError(msg)
E           ValueError: ExtensionArray contains a 'values' attribute that does not have a dtype attribute. This may cause issues in pandas' internals.

pandas/tests/extension/base/interface.py:64: ValueError
========================================================== 1 failed, 8 deselected in 0.12 seconds ===========================================================
```

What, if anything, should we include in pandas to test this? I'm fine with the current PR (no tests), since it's just test code.

Edit: ignore what I said about testing. Added one that calls `super()` inside a `raises` context manager. It's not too bad.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Closing in favor of #20794
DOC: add concatenation-section to text.rst
State: closed

As requested by @jreback in #20347.

Labels:
name: Docs
name: Strings
description: String extension data type and string data

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20790?src=pr&el=h1) Report
> Merging [#20790](https://codecov.io/gh/pandas-dev/pandas/pull/20790?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/0ae7e9090cee6e10ebb4124a0b7f3b30811ccc53?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/20790/graphs/tree.svg?height=150&src=pr&width=650&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/20790?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #20790      +/-   ##
==========================================
- Coverage   91.85%   91.84%   -0.01%     
==========================================
  Files         153      153              
  Lines       49308    49308              
==========================================
- Hits        45290    45288       -2     
- Misses       4018     4020       +2
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.23% <ø> (-0.01%)` | :arrow_down: |
| #single | `41.89% <ø> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/20790?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/strings.py](https://codecov.io/gh/pandas-dev/pandas/pull/20790/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3RyaW5ncy5weQ==) | `98.32% <ø> (ø)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/20790/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `84.59% <0%> (-0.21%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20790?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20790?src=pr&el=footer). Last update [0ae7e90...4ae9196](https://codecov.io/gh/pandas-dev/pandas/pull/20790?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

cc @jorisvandenbossche @TomAugspurger 
Build failure is not due to my commits (which do not touch the code base, just docs)
thanks
DEPR: removed long deprecated input param 'axis' in .replace()
State: closed

- [x] closes additional request in PR https://github.com/pandas-dev/pandas/pull/20271
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Deprecate
description: Functionality to remove in pandas

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20789?src=pr&el=h1) Report
> Merging [#20789](https://codecov.io/gh/pandas-dev/pandas/pull/20789?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/c4da79b5b322c73d8e61d1cb98ac4ab1e2438b40?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/20789/graphs/tree.svg?width=650&height=150&token=eZ4WkYLtcO&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/20789?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #20789      +/-   ##
==========================================
+ Coverage   91.79%    91.8%   +<.01%     
==========================================
  Files         153      153              
  Lines       49411    49409       -2     
==========================================
+ Hits        45359    45360       +1     
+ Misses       4052     4049       -3
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.2% <ø> (ø)` | :arrow_up: |
| #single | `41.91% <ø> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/20789?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/20789/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `95.98% <ø> (+0.04%)` | :arrow_up: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/20789/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.13% <ø> (ø)` | :arrow_up: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/20789/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `94.03% <ø> (ø)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/20789/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `84.59% <0%> (+0.2%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20789?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20789?src=pr&el=footer). Last update [c4da79b...c51213d](https://codecov.io/gh/pandas-dev/pandas/pull/20789?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

thanks @math-and-data