BUG: Preserve categorical dtypes when melting (#15853)
State: closed

This addresses loss of Categorical status for columns used as id_vars by following Jeff's suggestion to introduce tiling, so Index, Series, and Categorical all grew a `.tile` method.

- [X] closes #15853
- [X] tests added / passed
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Hello @dsm054! Thanks for submitting the PR.

 - There are no PEP8 issues in the file [`pandas/compat/numpy/function.py`](https://github.com/dsm054/pandas/blob/2f2eba0b5faa473aa781b55fed6b2a19c8a5e045/pandas/compat/numpy/function.py) !

 - There are no PEP8 issues in the file [`pandas/core/arrays/categorical.py`](https://github.com/dsm054/pandas/blob/2f2eba0b5faa473aa781b55fed6b2a19c8a5e045/pandas/core/arrays/categorical.py) !

 - There are no PEP8 issues in the file [`pandas/core/indexes/base.py`](https://github.com/dsm054/pandas/blob/2f2eba0b5faa473aa781b55fed6b2a19c8a5e045/pandas/core/indexes/base.py) !

 - There are no PEP8 issues in the file [`pandas/core/indexes/datetimelike.py`](https://github.com/dsm054/pandas/blob/2f2eba0b5faa473aa781b55fed6b2a19c8a5e045/pandas/core/indexes/datetimelike.py) !

 - There are no PEP8 issues in the file [`pandas/core/series.py`](https://github.com/dsm054/pandas/blob/2f2eba0b5faa473aa781b55fed6b2a19c8a5e045/pandas/core/series.py) !

 - There are no PEP8 issues in the file [`pandas/tests/arrays/categorical/test_analytics.py`](https://github.com/dsm054/pandas/blob/2f2eba0b5faa473aa781b55fed6b2a19c8a5e045/pandas/tests/arrays/categorical/test_analytics.py) !

 - There are no PEP8 issues in the file [`pandas/tests/indexes/test_base.py`](https://github.com/dsm054/pandas/blob/2f2eba0b5faa473aa781b55fed6b2a19c8a5e045/pandas/tests/indexes/test_base.py) !

 - There are no PEP8 issues in the file [`pandas/tests/reshape/test_melt.py`](https://github.com/dsm054/pandas/blob/2f2eba0b5faa473aa781b55fed6b2a19c8a5e045/pandas/tests/reshape/test_melt.py) !

 - There are no PEP8 issues in the file [`pandas/tests/series/test_analytics.py`](https://github.com/dsm054/pandas/blob/2f2eba0b5faa473aa781b55fed6b2a19c8a5e045/pandas/tests/series/test_analytics.py) !


# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23671?src=pr&el=h1) Report
> Merging [#23671](https://codecov.io/gh/pandas-dev/pandas/pull/23671?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/fe52d9f29416dd7d19c04ba7d47434b68c411bbd?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `95.45%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/23671/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/23671?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #23671      +/-   ##
==========================================
+ Coverage   92.24%   92.24%   +<.01%     
==========================================
  Files         161      161              
  Lines       51318    51340      +22     
==========================================
+ Hits        47339    47360      +21     
- Misses       3979     3980       +1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.63% <95.45%> (ø)` | :arrow_up: |
| #single | `42.3% <27.27%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/23671?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/23671/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `95.37% <100%> (+0.02%)` | :arrow_up: |
| [pandas/compat/numpy/function.py](https://codecov.io/gh/pandas-dev/pandas/pull/23671/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9udW1weS9mdW5jdGlvbi5weQ==) | `87.28% <100%> (+0.14%)` | :arrow_up: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/23671/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.46% <100%> (ø)` | :arrow_up: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/23671/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `93.72% <100%> (+0.04%)` | :arrow_up: |
| [pandas/core/indexes/datetimelike.py](https://codecov.io/gh/pandas-dev/pandas/pull/23671/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZWxpa2UucHk=) | `97.76% <83.33%> (-0.25%)` | :arrow_down: |
| [pandas/core/dtypes/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/23671/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbW1vbi5weQ==) | `94.37% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23671?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23671?src=pr&el=footer). Last update [fe52d9f...2f2eba0](https://codecov.io/gh/pandas-dev/pandas/pull/23671?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

I'm always hesitant to add new methods to Series / DataFrame. Do we think `.tile` is broadly useful enough to be included? `Series.tile` seems especially problematic, since you'll by definition end up with duplicate indices.

For fixing the original issue of preserving categorical in melt, I'd rather we define in internal helper function that knows how to tile extension arrays.

For Categorical we can tile the codes, datetimelike the `.asi8`, and for other EAs, we can convert to object, tile that, then use `_construct_from_sequence` to reconstruct the original dtype.
I don't understand your objection about the duplicate indices: while they're always awkward, that's exactly how pd.Series.repeat works already, and whose pattern I followed.

The objection to adding new methods is reasonable enough.
If you want to call it on the underlying values then it has to be part of
the EA interface, else it'll break for non-EAs.
And we have a pretty high bar for adding new methods to the interface.

On Wed, Nov 14, 2018 at 7:08 AM Jeff Reback <notifications@github.com>
wrote:

> *@jreback* requested changes on this pull request.
>
> allowing this method is ok, its basically another version of repeat.
> ------------------------------
>
> In pandas/core/series.py
> <https://github.com/pandas-dev/pandas/pull/23671#discussion_r233437629>:
>
> > @@ -1002,6 +1003,29 @@ def repeat(self, repeats, *args, **kwargs):
>          return self._constructor(new_values,
>                                   index=new_index).__finalize__(self)
>
> +    def tile(self, reps, *args, **kwargs):
> +        """
> +        Tile elements of a Series. Refer to `numpy.tile`
> +        for more information about the `reps` argument, although
> +        note that we do not support multidimensional tiling of Series.
> +
> +        See also
> +        --------
> +        pd.Series.repeat
> +        numpy.tile
> +        """
> +        nv.validate_tile(args, kwargs)
> +        new_index = self.index.tile(reps)
> +        if is_categorical_dtype(self.dtype):
> +            new_values = Categorical.from_codes(np.tile(self.cat.codes, reps),
>
> do this more like repeats, you just directly access the underlying
> function; then you don't need to type check
> ------------------------------
>
> In pandas/tests/indexes/test_base.py
> <https://github.com/pandas-dev/pandas/pull/23671#discussion_r233437753>:
>
> > @@ -2485,6 +2485,26 @@ def test_repeat(self):
>          result = index.repeat(repeats)
>          tm.assert_index_equal(result, expected)
>
> +    def test_tile(self):
>
> ideally you can use the indices fixture here and test all Index
> ------------------------------
>
> In pandas/tests/reshape/test_melt.py
> <https://github.com/pandas-dev/pandas/pull/23671#discussion_r233437895>:
>
> > +    @pytest.mark.parametrize('id_vars', [['a'], ['b'], ['a', 'b']])
> +    def test_categorical_id_vars(self, id_vars):
> +        # GH 15853
> +        df = DataFrame({"a": pd.Series(["a", "b", "c", "a", "d"],
> +                                       dtype="category"),
> +                        "b": pd.Series(pd.Categorical([0, 1, 1, 2, 1],
> +                                                      categories=[0, 2, 1, 3],
> +                                                      ordered=True)),
> +                        "c": range(5), "d": np.arange(5.0, 0.0, -1)},
> +                       columns=["a", "b", "c", "d"])
> +
> +        result = df.melt(id_vars=id_vars)
> +        for column in id_vars:
> +            num = len(df.columns) - len(id_vars)
> +            expected = df[column].tile(num).reset_index(drop=True)
> +            tm.assert_series_equal(result[column], expected)
>
> can you also test result itself
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/23671#pullrequestreview-174848506>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIgeuLCMpEJG6ww7cu-HXbO_zGWbMks5uvBXjgaJpZM4YcKNi>
> .
>

@TomAugspurger well this is actually pretty reasonable to add to be honest (and repeats too). Not adding things just causes more upstream pain.

I'm willing to take on some of that pain, if it means a cleaner API for EA authors.

We could write a repeat for EAs that operates uses either `ExtensionArray.take(np.repeat(np.arange(len(self))`, or a similar thing with factorized values.

I really don't think `.tile` makes sense for EAs, which are currently 1-D by definition.
> We could write a repeat for EAs that operates uses either ExtensionArray.take(np.repeat(np.arange(len(self)), or a similar thing with factorized values.

yes these could be implemented in terms of repeat, i agree.


can you rebase
@dsm054 can you merge master and update
can you merge master and update
closing as stale. pls ping if you want to continue working.