Melt enhance
State: closed

- [ x] closes #17676
- [ x] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
@tdpetrou I wouldn't go too far atm. This needs quite a lot of review. I am not real happy with all of these arguments.
Yea, the wide_to_long functionality can be nixed. It doesn't even exactly replicate it and then we'd be back to the same function signature
If you really wanted to incorporate wide_to_long functionality and minimize arguments, you could have a single extra parameter prefix that takes either a string or a two-item tuple with the first item being the separator.
@jreback Can we review this? This new melt function adds a ton of new functionality that I think would be very beneficial.
if you can rebase would be great 
@tdpetrou I would like to review this PR, but it is very difficult to see what has changed or what are the added enhancements to melt. Therefore, can you update the docstring, add a whatsnew note and add narrative docs in reshaping.rst ?
I agree with moving the implementation to a separate melt.py module, but for reviewing purposes I maybe would like to keep it where it was, and only afterwards move it (so not refactor + move at the same time).

> Yea, the wide_to_long functionality can be nixed. It doesn't even exactly replicate it and then we'd be back to the same function signature

For reviewing and discussing purposes, it might also be better to keep it in two separate PRs (the ability to melt on multiple columns (the referenced issue) and the introduction of wide_to_long similar functionality). But I don't know to what extent the implementation of the one is needed for the other? (how independent both additions are?)
This proposed API is very complicated. i would be much more in favor of something much simpler. Having a single function do a single thing is much much more intutive. A possible resolution is to introduce a lazy object with methods itself, akin to groupby. e.g.

```
r = df.reshape(.....)
r.melt()
r.to_long()
```

or somesuch, with each function performing 1 action
@jreback I am not a fan of that, as those are not really 'two-step' functions like you have with groupby/resample/rolling. So I don't see the point of doing something like that. If there is nothing really in common (apart from that it are reshaping methods), it should just be separate functions/methods (that could live in the same namespace, although in pandas we tend to either add it top-level or as method, so then already is in the one big namespace).

Let's focus here on the ability of handling multiple columns in melt, I don't think that part is that controversial, and if I am not wrong (but didn't look yet in detail) does not add new keywords (is that correct @tdpetrou ?)
To first address the two-step function. I agree with @jorisvandenbossche and don't think that is necessary. Although, the `wide_to_long` functionality can be captured with a single additional parameter, let's keep things simple and just drop it completely from this PR. 

So, I will:

- Keep melt in its original file reshape.py
- Remove the wide to long functionality - meaning no extra parameters
- Change the whatsnew
- Add notes to reshaping.rst
@tdpetrou can you take a look at https://github.com/pandas-dev/pandas/pull/17459 and see how those would conflict / overlap with eachother?

I haven't had a chance to go through the changes here in detail yet.
I think everything is updated and rebased appropriately. I had to delete a few changes #17628 that were mixed in here. I made a little notebook on the enhancement as well (https://nbviewer.jupyter.org/github/tdpetrou/Machine-Learning-Books-With-Python/blob/master/melt%20enhancemenet.ipynb)

@TomAugspurger I don't think it will be much of a problem to keep the index. It'll just be like another entry to`id_vars`

One more thing: There are two test failures in the wide_to_long test class. This is because my new melt function treats `None` and an empty list as the same thing when passed to either `value_vars` or `id_vars`. The current melt function will return an empty dataframe if you pass an empty list to `value_vars`. However, if you pass an empty list to `id_vars` then it returns the same thing as if it were `None`.
so, this is pretty much impossible to review as is. I would first do a simple PR which moves the implementation of .melt to a separate module. This would be a simple copy/paste. Then you can make this implementation much more simple, but for example not in-lining functions inside melt itself, rather defining them in the module.


@jreback 
I ....
* put melt in its own module
* removed any nested functions
* made a separate tests module
* added a couple examples to whatsnew
* simplified an example and added one in the reshaping docs
@jreback Getting back to this now. I put melt in its own module as requested #18148 
Hello @tdpetrou! Thanks for updating the PR.







Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on December 10, 2017 at 22:57 Hours UTC
@jreback I just rebased this and would like to work on it
ok. this is doing a lot of things. I would appreciate to break this apart into separate distinct changes. The code needs some work. I can comment, but easier to do it in smaller pieces.
@jreback There is basically one piece, now that the wide_to_long functionality has been removed. Essentially, you pass a list of lists into `value_vars` to simultaneously melt those sets of columns. 

I suppose you could break it up into 3 pieces
* Make only original `melt` work with my new way
* Add in MultiIndex support - this handles all varieties of string/integer MultiIndex
* Finally add in simultaneously melt

I think I can break it up like this without too much effort. What do you think?
closing as stale. ping if you want to update.