Preserve EA dtype in DataFrame.stack
State: closed

closes #23077

There were two bugs in master (not present in 0.23.4), probably from the SparseArray PR

1. We need to unbox the EA values from Series before passing to `EA._concat_same_type`
2. We need to followup with a `take` to get the correct order.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Just a WIP for now. I feel like we're lacking on tests here. Will add one for stacking a single level from a dataframe with a MultiIndex in the columns.
Hello @TomAugspurger! Thanks for updating the PR.

 - There are no PEP8 issues in the file [`pandas/core/internals/blocks.py`](https://github.com/TomAugspurger/pandas/blob/d6661cb6490916b808b2f9c0ac650bec56ad8cf7/pandas/core/internals/blocks.py) !

 - There are no PEP8 issues in the file [`pandas/core/reshape/reshape.py`](https://github.com/TomAugspurger/pandas/blob/d6661cb6490916b808b2f9c0ac650bec56ad8cf7/pandas/core/reshape/reshape.py) !

 - There are no PEP8 issues in the file [`pandas/tests/extension/base/reshaping.py`](https://github.com/TomAugspurger/pandas/blob/d6661cb6490916b808b2f9c0ac650bec56ad8cf7/pandas/tests/extension/base/reshaping.py) !

 - There are no PEP8 issues in the file [`pandas/tests/extension/json/test_json.py`](https://github.com/TomAugspurger/pandas/blob/d6661cb6490916b808b2f9c0ac650bec56ad8cf7/pandas/tests/extension/json/test_json.py) !

 - There are no PEP8 issues in the file [`pandas/tests/frame/test_reshape.py`](https://github.com/TomAugspurger/pandas/blob/d6661cb6490916b808b2f9c0ac650bec56ad8cf7/pandas/tests/frame/test_reshape.py) !

 - There are no PEP8 issues in the file [`pandas/tests/sparse/frame/test_frame.py`](https://github.com/TomAugspurger/pandas/blob/d6661cb6490916b808b2f9c0ac650bec56ad8cf7/pandas/tests/sparse/frame/test_frame.py) !



##### Comment last updated on October 24, 2018 at 21:13 Hours UTC
"Fixed" the sparse values. We were failing to handle `DataFrame[SparseArray].astype(object)` correctly. On master, Series[sparse].astype(object) / Frame[sparse].astype(object) is sparse, but I think we want to change that. https://github.com/pandas-dev/pandas/issues/23125

I'll probably do that today, and then pick up the reshaping PRs afterwards.
I think we need tests where the columns are a MultiIndex.

________________________________
From: Joris Van den Bossche <notifications@github.com>
Sent: Tuesday, October 23, 2018 4:24:31 PM
To: pandas-dev/pandas
Cc: Tom Augspurger; Mention
Subject: Re: [pandas-dev/pandas] [WIP]Preserve EA dtype in DataFrame.stack (#23285)


@jorisvandenbossche commented on this pull request.

Is this still WIP ?
It looks good to me.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/pandas-dev/pandas/pull/23285#pullrequestreview-167643508>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABQHIqd_gEfs-HuVEgu9F_eRaTpryWp9ks5un4kPgaJpZM4X0JPS>.

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23285?src=pr&el=h1) Report
> Merging [#23285](https://codecov.io/gh/pandas-dev/pandas/pull/23285?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/82120016e6af85afb052d95710e498a6dc58c1ce?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/23285/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/23285?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #23285      +/-   ##
==========================================
+ Coverage   92.24%   92.25%   +<.01%     
==========================================
  Files         161      161              
  Lines       51224    51237      +13     
==========================================
+ Hits        47254    47269      +15     
+ Misses       3970     3968       -2
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.63% <100%> (ø)` | :arrow_up: |
| #single | `42.27% <5.88%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/23285?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/internals/blocks.py](https://codecov.io/gh/pandas-dev/pandas/pull/23285/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2Jsb2Nrcy5weQ==) | `93.67% <ø> (ø)` | :arrow_up: |
| [pandas/core/reshape/reshape.py](https://codecov.io/gh/pandas-dev/pandas/pull/23285/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9yZXNoYXBlLnB5) | `99.56% <100%> (+0.01%)` | :arrow_up: |
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/23285/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `95.34% <0%> (+0.25%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23285?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23285?src=pr&el=footer). Last update [8212001...f6aeafa](https://codecov.io/gh/pandas-dev/pandas/pull/23285?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

can you rebase
All green.
Merged master to fix the merge conflict with the unstack PR. Will ping on green.
lgtm.