Initial draft: from_dummies
State: closed

Converts dummy codes to categorical variables.

- [x] Closes [#8745](https://github.com/pandas-dev/pandas/issues/8745)
- [x] Passes atomic options tests
- [x] Passes linting tests
- [x] Whatsnew entry

Tests will be expaneded with composite options and edgecases as soon as definition of final scope is complete.
~~I tried to mirror the `get_dummes` function wherever possible to ensure an as close inverse of the function as possible~~

Primary goal of the draft is to demonstrate the options I had in mind, optimization is the next step as soon as we decided what we want to keep, cut, or add. 
(Docstrings etc. will be added based on results)

Integration as method of `categorical`, as proposed in [#34426](https://github.com/pandas-dev/pandas/pull/34426), can be considerd as a next step if moving in that direction is planed.

## Current Options:

### no arguments
- Assumes all columns to be cateogries of the same variable which is called `categories`
    - example: 
        ```python
        >>> dummies
              a   b   c   d
        0     1   0   0   0      
        1     0   1   0   0      
        2     1   0   0   0
        3     0   0   0   1      
        4     0   0   1   0         

        >>> from_dummies(dummies)
            categories
        0       a      
        1       b       
        2       a
        3       d
        4       c     
        ```

### subset **-- [RMOEVED](https://github.com/pandas-dev/pandas/pull/41902#discussion_r758820277) --**
-  Select which columns of the input `DataFrame` to include for decoding.
- `None`:
    - Assumes the entire passed `DataFrame` consists of dummy coded variables
- `List` or `Index`:
    - Just returns the decoded subset passed in the list. 
    - example: 
        ```python
        >>> dummies
           C    col1_a  col1_b  col2_a  col2_b  col2_c
        0  1       1       0       0       1       0
        1  2       0       1       1       0       0
        2  3       1       0       0       0       1

        >>> from_dummies(dummies, sep="_", subset=["col1_a", "col1_b", "col2_a", "col2_b", "col2_c"])
          col1    col2 
        0  a       b
        1  b       a 
        2  a       c
        ```

### variables **-- [REMOVED](https://github.com/pandas-dev/pandas/pull/41902#discussion_r650566136) --**
- Inverts the `prefix` argument of `get_dummies`
- Using: 
    ```python
    >>> dummies
      col1_a  col1_b  col2_a  col2_b  col2_c
    0    1       0       0       1       0
    1    0       1       1       0       0
    2    1       0       0       0       1
    ``` 
- `None`:
    - Variable names are taken from prefix.
    - example: 
        ```python
        >>> from_dummies(dummies, sep="_", variables=None)
          col1    col2 
        0   a       b
        1   b       a 
        2   a       c
        ```
- `str`:
    - Variable names are numbered from the passed string.
    - example:
        ```python
        >>> from_dummies(dummies, sep="_", variables="Varname")
           Varname1   Varname2 
        0     a          b
        1     b          a 
        2     a          c
        ```
- `List`:
    - Variable names are obtained (in order) from the passed list.
    - example:
        ```python
        >>> from_dummies(dummies, sep="_", variables=["One", "Two"])
            One     Two 
        0    a       b
        1    b       a 
        2    a       c
        ```
- `Dict`:
    - Variable names are obtained from passed prefix to variable mapping.
    - Currently orders the returned columns by index in the passed dictionary, not sure if that should be changed to strictly follow the order of prefixes in the dummy-data.
    - example:
        ```python
        >>> from_dummies(dummies, sep="_", variables={"col2": "One", "col1": "Two"})
            One     Two   
        0    b       a
        1    a       b
        2    c       a
        ```

### sep
- Separator used in the column names of the dummy categories they are character indicating the separation of the categorical names from the prefixes. For example, if your column names are `prefix_A` and `prefix_B`, you can strip the underscore by specifying `sep='_'`.
- Required argument if prefixes are to be separated as there is no default separator.
- `str`:
    - Splits columns by first instance of passed string to obtain variables and elements.
    - example: 
        ```python
        >>> dummies
           col1_a  col1_b  col2_a  col2_b  col2_c
        0    1       0       0       1       0
        1    0       1       1       0       0
        2    1       0       0       0       1
 
        >>> from_dummies(dummies, sep="_")
           col1    col2 
        0   a       b
        1   b       a 
        2   a       c
        ```
- `list`: **-- [REMOVED](https://github.com/pandas-dev/pandas/pull/41902#discussion_r744318647) --**
    - Splits columns by first instance of any of the seperators passed in the list.
    - example:
        ```python
        >>> dummies
          col1_a  col1_b  col2-a  col2-b  col2-c
        0   1       0       0       1       0
        1   0       1       1       0       0
        2   1       0       0       0       1
        
        >>> from_dummies(dummies, sep=["_", "-"]) 
           col1    col2 
        0   a       b
        1   b       a 
        2   a       c
        ```
- `dict`: **-- [REMOVED](https://github.com/pandas-dev/pandas/pull/41902#discussion_r758820277) --**
    - Splits columns with prefix by first instance of corresponding seperator.
    - example: 
        ```python
        >>> dummies
           col1_a-a  col1_b-b  col_2-a   col_2-b   col_2-c
        0     1         0         0         1         0
        1     0         1         1         0         0
        2     1         0         0         0         1

        >>> from_dummies(dummies, sep={"col1": "_", "col_2": "-"})
            col1     col_2 
        0    a-a       b
        1    b-b       a 
        2    a-a       c
        ```

### dummy_na **-- [REMOVED](https://github.com/pandas-dev/pandas/pull/41902#discussion_r652048588) --**
-  mirrors the `dummy_na` argument of `get_dummies`
- `False` but contains NaN:
    - Considers row with no assignments as NaN.
    - example: 
        ```python
        >>> dummies
           col1_a  col1_b  col2_a  col2_b  col2_c
        0    1       0       0       0       0
        1    0       1       1       0       0
        2    0       0       0       0       1

        >>> from_dummies(dummies, sep="_", dummy_na=False)
            col1    col2 
        0    a      NaN
        1    b       a 
        2   NaN      c
        ```
- `True`:
    - Considers row with no assignments as NaN.
    - example: 
        ```python
        >>> dummies
           col1_a  col1_b  col1_NaN  col2_a  col2_b  col2_c col2_NaN
        0    1       0        0        0       0       0       1
        1    0       1        0        1       0       0       0
        2    0       0        1        0       0       1       0

        >>> from_dummies(dummies, sep="_", dummy_na=True)
           col1    col2 
        0   a      NaN
        1   b       a 
        2  NaN      c
        ```

### base_category
- Inverts the `drop_first` argument of `get_dummies`
- Using: 
    ```python
    >>> dummies
       col1_a  col1_b  col2_a  col2_b  col2_c
    0     1       0       0       0       0
    1     0       1       1       0       0
    2     0       0       0       0       1
    ``` 
- `None`: **-- [REMOVED](https://github.com/pandas-dev/pandas/pull/41902#discussion_r683637232) --**
    - Without `dummy_na=True` non-asigned rows are considered NaN.
    - example:
        ```python
        >>> from_dummies(dummies, sep="_", base_category=None)
           col1    col2 
        0    a      NaN
        1    b       a 
        2   NaN      c
        ```
- `str`:
    - Assumes all missing assignments to be the base category of the passed category.
    - example:
        ```python
        >>> from_dummies(dummies, sep="_", base_category="x")
           col1    col2 
        0    a       x
        1    b       a 
        2    x       c
        ```
- `List`: **-- [REMOVED](https://github.com/pandas-dev/pandas/pull/41902#discussion_r744318041) --**
    - The base categories for variables are obtained (in order) from the passed list.
    - example:
        ```python
        >>> from_dummies(dummies, sep="_", base_category=["x", "y"])
           col1    col2 
        0    a       y
        1    b       a 
        2    x       c
        ```
- `Dict`:
    - The base categories for variables are obtained from the passed prefix to base category mapping.
    - example: 
        ```python
        >>> from_dummies(dummies, sep="_", base_category={"col2": "x", "col1": "y"})
           col1    col2 
        0    a       x
        1    b       a 
        2    y       c
        ```
       
### fillna **-- [REMOVED](https://github.com/pandas-dev/pandas/pull/41902#discussion_r663350073) --**
- Using: 
    ```bash
    >>> dummies
       col1_a  col1_b  col2_a  col2_b  col2_c
    0     1       0       0      NaN      0
    1     0       1       1       0       0
    2     0      Nan      0       0       1
    ``` 
- `True`:
    - Can result in double assignemnt, which will raise the respective error.
    - example:
        ```bash
        >>> from_dummies(dummies, sep="_", fillna=True)
           col1    col2 
        0    a       b
        1    b       a 
        2    b       c
        ```
- `False`:
    - Can result in unassigned row which will be treated as any other unassigned row.
    - example:
        ```bash
        >>> from_dummies(dummies, sep="_", fillna=False)
           col1    col2 
        0    a      NaN
        1    b       a 
        2   NaN      c
        ```

## To Discuss:
1. Change, remove, or add options?


## To-Do
- [x] Add handling of NA values in input
- [ ] Adapt discussed changes ([~~C1~~](https://github.com/pandas-dev/pandas/pull/41902#discussion_r650566136), [~~C2~~](https://github.com/pandas-dev/pandas/pull/41902#discussion_r652048588))
- [ ] Expand tests
- [ ] Work on optimization ([O1](https://github.com/pandas-dev/pandas/pull/41902#discussion_r652048104))
- [ ] Raise relevant errors
- [ ] Whatsnew entry



Labels:
name: Enhancement
name: Categorical
description: Categorical Data Type

Comments:
Wow, some serious effort here!

The examples you've put in the body of the PR are kind of hard to follow - perhaps if you write them from a Python REPL they're easier to understand? 

e.g.:
```python
>>> dummies
   col_a  col_b  col_c
0      1      0      0
1      0      1      0
2      0      0      1
3      1      0      0
>>> pd.from_dummies(dummies)
   col 
0    a
1    b
2    c
3    a
```
This is still marked as "draft" - just checking, in case you think it's ready for review
> This is still marked as "draft" - just checking, in case you think it's ready for review

Good point, I will write the documentation etc. and add more tests for the features we decided to keep, then I will set it ready for review.
There are still some work in progress points, such as user guide and more tests etc., but these will incorporate the incoming feedback. 
Current state of `dummies` with mixed columns (with and without separator) - based on [those thoughts](https://github.com/pandas-dev/pandas/pull/41902#discussion_r724242484):
```python
>>> dummies = pd.DataFrame(
        {
            "a": [1, 0, 0], 
            "b": [0, 1, 0], 
            "c": [0, 0, 1],
            "col1_a": [1, 0, 0],
            "col1_b": [0, 1, 0],
            "col2_a": [0, 1, 0],
            "col2_b": [0, 0, 0],
            "col2_c": [0, 0, 1],
        },
    )
 ```
Raises an error (`ValueError: Separator not specified for all columns; First instance column: a`) when attempting to do the following:
```python
>>> from_dummies(dummies, sep="_")
```
or
```python
>>> from_dummies(dummies, sep={"col1": "_", "col2": "_"})
```
I think manually splitting the `dummies` and converting the subsets, with and without separator, individually (or just using the `subset` argument) is more tracable for a user (Compared to having the function automatically splitting the input).

Yes, exactly.

On Sun, Jan 9, 2022, 16:02 Peter ***@***.***> wrote:

> ***@***.**** commented on this pull request.
> ------------------------------
>
> In pandas/tests/reshape/test_from_dummies.py
> <https://github.com/pandas-dev/pandas/pull/41902#discussion_r780799917>:
>
> > @@ -0,0 +1,313 @@
> +import numpy as np
>
> With round trip you mean dummify a categorical with get_dummies then
> reverse it with from_dummies?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/41902#discussion_r780799917>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABKTSRO22E4PSKZDSGPD3BDUVGWRZANCNFSM46MN3P6A>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you commented.Message ID:
> ***@***.***>
>

Is there anything else I should add or change?
pls merge master 
> pls merge master

Done, but I do not quite understand the Doc Build error as it does not appear to be in any file that I touched and, with the exception of merging main, I did not change anything since [015ee94](https://github.com/pandas-dev/pandas/pull/41902/commits/015ee94ce21836e19ce0e7e452bae4e12940a210) which passed all checks.
@pckSF some doc-string validation issues: https://github.com/pandas-dev/pandas/runs/6738570255?check_suite_focus=true
> exception: Error parsing See Also entry '``DataFrame`` of dummy variables.' in the docstring of get_dummies in /home/runner/work/pandas/pandas/pandas/core/reshape/encoding.py.)

still something with the doc build
> > exception: Error parsing See Also entry '`DataFrame` of dummy variables.' in the docstring of get_dummies in /home/runner/work/pandas/pandas/pandas/core/reshape/encoding.py.)
> 
> still something with the doc build

Jep, it seems like I always break something when trying to fix something else ... I will stay on it until everything is green.
> > > exception: Error parsing See Also entry '`DataFrame` of dummy variables.' in the docstring of get_dummies in /home/runner/work/pandas/pandas/pandas/core/reshape/encoding.py.)
> > 
> > 
> > still something with the doc build
> 
> Jep, it seems like I always break something when trying to fix something else ... I will stay on it until everything is green.

:-> also can try to build that page locally and isolate things
@bashtage @MarcoGorelli @fangchenli When you have the chance to review and approve if all looks good, it would be appreciated.
> .loc[:, prefix_slice]

I just changed that real quick to be on the safe side, also thanks for your and the teams patience :)
thanks @pckSF really nice