ENH: add expand kw to str.get_dummies
State: closed

Ref: #10008. 

Though this still under work (needs #10089 to simplify `get_dummies` flow), would like to discuss followings.

~~#### `.str.extract` note: overlaps with #11386~~

~~Currently it returns `Series` for a single group and `DataFrame` for multiples. To support `expand` kw, we have to choose :~~

~~1. Add `expand` option keeping existing behavior with warning for future change to `extract=True` (current impl).~~
~~2. Add `expand` option keeping existing behavior. Standardize `extract=None` (or other option) to select returning dimensionality automatically.~~ 
~~3. Add `expand` option with default `True` (or `False`). This breaks the API.~~
~~4. Make `Index.str.extract` return `MultiIndex` in multiple group case without adding `expand` option.~~
#### `.str.get_dummies`
1. Add `expand` kw with default `True`. Currently this always returns `DataFrame` (and raises `TypeError` in `Index`). This doesn't break an API (current impl).
2. Make `Index.str.get_dummies` return `MultiIndex` without adding `expand` option.

CC @mortada


Labels:
name: API Design
name: Strings
description: String extension data type and string data
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Question: it there actually a need to have the option of `expand` here for those two functions? 

If we add them, my opinion about the discussion points:
- `extract`:
  - I would keep the current behaviour (seems also as a good behaviour (series for single group, dataframe for multiple groups), I see no reason to change?)
  - Have `expand=None` to let you able to override the default behaviour
- `get_dummies`:
  - OK. How would this look for `expand=False`? A series/index with lists as elements? (question is if we want to encourage this? -> coming back to my initial question)

@jorisvandenbossche Correct. There is an option to make `Index.str` work as the same as `Series.str` without adding `expand` kw. Added as alternatives. Though I prefer to make them to have unified kw/behavior.

> `get_dummies`: How would this look for expand=False? A series/index with lists as elements?

Result will be a `Series`/`Index` of tuples, as the same as `str.split(expand=False)`.

Not that important for this discussion, but `str.split` gives a list, not a tuple for me?

> Though I prefer to make them to have unified kw/behavior.

Well, I would also like that very much, but the default values of the keyword would in any case _not_ be unified. So therefore, as it is not really possible to unify it that way, I was considering the option of not adding the keyword at all (which is also no unified behaviour)

status?

I hope to work on this, but it requires `Index.fillna` to make flow simple (#10089). 

@sinhrks what are we doing with this one?

There are 2 points, and I think 1st point (add `expand=False` to return `Series`) is less useful (only for consistency). How about adding `Index.str.get_dummies` and close the issue?
1. Add `expand` kw with default `True`. Currently this always returns `DataFrame` (and raises `TypeError` in `Index`). This doesn't break an API (current impl).
2. Make `Index.str.get_dummies` return `MultiIndex` without adding `expand` option.

@sinhrks I suppose you could add `Index.str.get_dummies` not really how useful this is, but it makes things consistent.