Categorical.(get|from)_dummies
State: closed

Simply converting categorical variables to and from dummy variables.

- [x] closes #8745
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Intentionally smaller-scoped than #31795 (and indeed `get_dummies`) as a broadly useful MVP which can be chained with other basic functionality. The tests are fairly rudimentary and I welcome any edge cases which should be picked out.

## For discussion

### from_dummies

* class method rather than free function: Keeps categorical-related functionality together, reduces surface in the global namespace, more obvious what is produced.
* silently drop column with NA header: wasn't sure about this. Maybe it should raise a warning?
* No handling of masked dataframes or dataframes with NA values
* No subsetting or renaming of columns: callers can do this themselves

### to_dummies

* Name: I went for `Categorical.to_dummies` instead of matching the free function `get_dummies`. The symmetry of to/from aids understanding, and using `get_` might imply A) that something is being got, which it isn't or B) signature/ feature parity with the existing method, which wasn't a design goal for me.
* `to_dummies` return type: `cls.to_dummies` returns bools, where `get_dummies` returns uint8s by default, which doesn't make a lot of sense to me as we are representing boolean data (and they're the same in memory anyway). Dummy variables are generally used for regression where a continuous variable is required, so ints don't get us any closer to what we may want, and being able to index into the categories using the row may be valuable.
* No `dtype` argument: I didn't see any benefit of `cls.to_dummies(dtype=float)` over `cls.to_dummies().astype(float)`. The latter is more explicit, no slower, and minimises API surface.
* No `prefix`, `prefix_sep` arguments: these unnecessarily assume string column headers. If someone wants to rename their columns, they can: IMO it's not a core requirement of this method.
* `dumma_na` replaced with `na_column`: if we're including the argument, we may as well let the user decide what they want to call their column, using the dtype they prefer (e.g. `"other"`, `-1` etc). They can always supply `np.nan` if they want `get_dummies`-like behaviour.
* No `sparse` argument: This I regret. Producing a sparse array would be valuable. However, it would drastically complicate the method, so I left it out for the MVP. The caller can always sparsify it after it's produced, so long as RAM isn't an issue for the temporary df.
* No `drop_first` argument: If someone wants to drop one of their columns, they can (`cls.to_dummies().drop(columns="my_col")`): again, not a core requirement of this method. Broadly speaking, I'm not in favour of adding arguments to save the caller <=1 line of their own code unless there are e.g. speed gains.

Labels:
name: Enhancement
name: Categorical
description: Categorical Data Type
name: Stale

Comments:
likely would want a doc-example of both of these near in https://pandas.pydata.org/docs/dev/user_guide/reshaping.html#computing-indicator-dummy-variables and/or links to categorical.rst
> I think this should be called get_dummies I don't think the justification for deviating from that name is strong enough to warrant a different name.
>
> I also think it should exactly match the signature of get_dummies.

@TomAugspurger My initial implementation was a thin wrapper around `get_dummies` but given how simple the "raw" implementation is I didn't see a huge benefit to it. I could switch back, although I had wondered whether it might be better for for `get_dummies` to actually call `to_dummies` in some cases to avoid some logic.

I do feel more strongly about keeping the name and API different to `get_dummies`, though. `get_` methods imply retrieving something from the object rather than converting it to a different form. `from_` is IMO the most obvious name for the alternate constructor, and if `from_` exists, the opposite ought to be `to_` (`as_` or `into_` would also be OK but I feel they imply more mutation/ destructive operations). I personally have a minor nit with the naming of pandas' IO methods on this basis - `read_csv` and `to_csv` are from two different opposite pairs, which increased the time it took for me to remember them.

My justifications for not supporting all of the `get_dummies` interface are in the PR description. Supporting the whole `get_dummies` API in `to_dummies` would necessitate supporting all of the reverse operations in `from_dummies` (which is the key part of this PR; `to_dummies` is more of an afterthought), which I considered to be a large complexity overhead for not a lot of gain in utility. If we're not supporting the whole API, they shouldn't have the same name.

Would be interested for other parties to weigh in.
> I do feel more strongly about keeping the name and API different to get_dummies

Yeah, your reasons for `to_dummies` all make sense. But in my mind it comes down to a values judgement between

1. Symmetry the two methods, which favors `to_dummies` over `get_dummies`, as `to_dummies` aligns better with `from_dummies`
2. Consistency with the established name `get_dummies`.

In my opinion, 2 wins out.

> Supporting the whole get_dummies API in to_dummies would necessitate supporting all of the reverse operations in from_dummies

That would indeed be helpful. Which parameters / behaviors are especially hard to support? I'm happy to raise a `NotImplementedError` for things that could be done in a followup PR if anyone is interested.
@clbarnes Is this still active? Can you fix merge conflicts if so?
Sorry I haven't been able to get back to this, I'll see what I can do today.

The tl;dr of the changes requested seems to be

- [x] Dummy creation class method should be called `get_dummies`
  - [x] This should be a thin wrapper around the `get_dummies` function and have an identical signature
- [x] Better cross-references between categorical and get_dummies docs
- [ ] See above docstring requests
- [x] Rebase
I'm calling `pandas.get_dummies` in `Categorical.get_dummies`. I'd rather be calling `pandas.core.reshape.reshape._get_dummies_1d`, but that fails a lint, and I don't want to start renaming existing functions - any guidance on that?
Still to do:

- [ ] Edge cases for np.nonzero with "boolean" dtype in Categorical.from_dummies
- [ ] More tests, generally
- [ ] Check cohesiveness of docs
Hello @clbarnes! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-22 15:22:56 UTC
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@clbarnes how's this going? If you're interested in continuing can you merge master & resolve conflicts and address comments?
Sure, will do - did the first 90% but things got busy before I could tackle the last 90%. I'll try to get to this next week.
> Sure, will do - did the first 90% but things got busy before I could tackle the last 90%. I'll try to get to this next week.

Sounds good (& no rush ofc).  Ping us when you're ready for the next round of reviews
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@clbarnes still active? I think this was pretty close.
Thanks for the PR, but it appears to have gotten stale. Going to close for now but let us know if you can merge master and update and we'd be happy to reopen.