ENH: Add new implementation of DataFrame.stack
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

We can either deprecate the current implementation in 2.1, or wait until 2.2. It's slightly more gradual to wait until 2.2; users in 2.1 could opt into the new behavior without any warnings.

Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
How do you see the deprecation/update process? If we don't yet warn in 2.1, I doubt that many users will already try out `v3=True`. And for warning, do you think it would be possible to detect _when_ to warn? Or were you planning to warn for all cases to say that people should specify v3=True to get rid of the warning? (ideally we would only warn if v3 would actually give a different result, otherwise it will be a noisy annoying warning for those (the majority?) use cases that don't need it) 
Instead of a v3 keyword could use the "future" option from #53025
@jorisvandenbossche 

> How do you see the deprecation/update process? If we don't yet warn in 2.1, I doubt that many users will already try out `v3=True`. And for warning, do you think it would be possible to detect _when_ to warn?

I was thinking we add the deprecation in 2.2; I agree not many will use `v3=True` in 2.1, but I don't see any negatives of waiting until 2.2 to deprecate.

For doing the deprecation, we could get really complicated. I think behavior is the same when:

 - stacking where columns are a non-MultiIndex;
 - stacking where columns are a MultiIndex and every combination between stacked and non-stacked levels occurs and `dropna=False`;
 - stacking where columns are a MultiIndex and every combination between stacked and non-stacked levels occurs and `dropna=True` and the input does not have any NAs; or
- stacking where columns are a MultiIndex, dtypes are all float, dropna=True, and the input does not have NaN values

However, since we'd be deprecating `dropna` we'd still need to warn in the 2nd bullet above.

@jbrockmendel 

> Instead of a v3 keyword could use the "future" option from #53025

I'd definitely be on board with this.
@jbrockmendel @jorisvandenbossche @mroeschke assuming the CI agrees I think this is ready for review.

@jbrockmendel - regarding using a `future.*` option here, I think I'd prefer the additional argument since this is just on a single method. But if we're wanting something like "all opting into future behaviors is done via `future.*` options", I'd be on board with that and happy to rework that here. Thoughts?
>  But if we're wanting something like "all opting into future behaviors is done via future.* options"

Definitely not
Thanks @mroeschke - I think this is good for another look.
Awesome, thanks @rhshadrach