Repository: pyjanitor-devs/pyjanitor

Issue:
Deprecate label_encode?
I recently stumbled upon pandas' function `factorize`. I think it does what we want with `label_encode`, except it is not a dataframe method.

I'd like to propose the following course of action:

1. First off, switch out the label_encode implementation to use pandas' factorize.
2. Add an alias function `factorize` that enables this to be a dataframe method.
3. Add a warning that we will be deprecating `label_encode`.
4. Deprecate label_encode in the 1.x release.

What are everybody's thoughts on this one?


Discussion:
@ericmjl @samukweku Is this being worked on by anyone ? If not, I would like to take this up.  Thanks.
@nvamsikrishna05 pls go ahead.
@samukweku @ericmjl  Thanks

Just want to write down the thoughts here. Please correct me if I'm wrong.
(1)
The Current Solution - 
``` python
def label_encode(
    df: pd.DataFrame, column_names: Union[str, Iterable[str], Hashable]
) -> pd.DataFrame:
    df = _factorize(df, column_names)
    return df

@dispatch(pd.DataFrame, (list, tuple))
def _label_encode(df, column_names):
    le = LabelEncoder()
    check_column(df, column_names=column_names, present=True)
    for col in column_names:
        df[f"{col}_enc"] = le.fit_transform(df[col])
    return df

@dispatch(pd.DataFrame, str)  # noqa: F811
def _label_encode(df, column_names):  # noqa: F811
    le = LabelEncoder()
    check_column(df, column_names=column_names, present=True)
    df[f"{column_names}_enc"] = le.fit_transform(df[column_names])
    return df

```

Solution I'm thinking of with factorize would be - 

``` python

@dispatch(pd.DataFrame, (list, tuple))
def _factorize(df, column_names):
    check_column(df, column_names=column_names, present=True)
    for col in column_names:
        df[f"{col}_enc"] = pd.factorize(df[col])[0]
    return df

@dispatch(pd.DataFrame, str)
def _factorize(df, column_name):
    check_column(df, column_names=column_name, present=True)
    df[f"{column_name}_enc"] = pd.factorize(df[column_name])[0]
    return df
```
These two would be replacing the _label_encode implementations.

The main benefit of doing this would be we would be wrapping over the pandas implementation and can remove the sklearn LabelEncoder. I see that sklearn is directly used just for this particular function only. so we can remove it ?

(3) For adding a warning that label_encode is this the right approach or are we using something else ?
``` python
 warnings.warn("label_encode will be deprecated in future release. Please use factorize", DeprecationWarning)
```
This would be at the start of label_encode

(2) Add an alias factorize -  another function like
``` python
def factorize(df, column_name):
    pass
```
In this way like a function overloading over the pandas implementation ? Wouldn't that conflict over pandas one ?

Couple of questions I have -
The pandas factorize implementation has optional parameter -
 na_sentinel=- 1 # used to convert NaN. Default is -1 which can be changed.
sort = False  # used to sort the unique value and then assign the numbering.
Should we be providing this option ? LabelEncoder doesn't have it so we didn't have it so far.

Hi @nvamsikrishna05! Thanks for thinking through this issue so thoroughly. I am going to answer your questions one by one.

> I see that sklearn is directly used just for this particular function only. so we can remove it ?

Yes! That is correct; it's always nice to minimize the number of dependencies of a package, especially big ones. 

> For adding a warning that label_encode is this the right approach or are we using something else ?

The implementation you suggested looks about correct.

> In this way like a function overloading over the pandas implementation ? Wouldn't that conflict over pandas one ?

You are also correct here. What do you think of providing a `factorize_columns` function instead? I'm thinking of this because one value add `pyjanitor` provides over `pandas` is to make convenient some pandas operations, i.e. 

> The pandas factorize implementation has optional parameter -
> na_sentinel=- 1 # used to convert NaN. Default is -1 which can be changed.
> sort = False # used to sort the unique value and then assign the numbering.
> Should we be providing this option ? LabelEncoder doesn't have it so we didn't have it so far.

It makes sense to pass-through keyword arguments from `factorize_columns` into `factorize`. I think it's sensible to start simple first and assume that `factorize` keyword arguments are applied uniformly across all columns on which we are applying factorize, unless there's a better pattern that we can follow.

The pattern here would look akin to:

```python
def factorize_columns(df, column_names, **factorize_kwargs):
    """Docstring goes here."""
    for column in column_names:
        df[f"{column}_enc"] = pd.factorize(df[column], **factorize_kwargs)
    return df
```
@ericmjl Thanks for the clarifications.  Yes, factorize_columns name sounds good. Also should the keyword arguments be added to the to-be deprecated label_encode function also ?

About retiring the label_encode function, doesn't "label_encode" name is more meaningful for people looking to do Label Encoding quickly ? Is the reasoning behind using factorize_columns so that people understand that this is the convenience method over the pandas factorize (mind map linking 1-1) ?
@nvamsikrishna05 you raise a great question there! Thanks for your patience, btw, I was catching up with life after doing my SciPy tutorial on Tuesday. I also took the time to ponder over your question on-and-off.

Indeed, because `pyjanitor` is really about convenience functions, I think there are two ways we could proceed here. We could replace just the `label_encode` implementation, with the effect of removing `pyjanitor`'s dependency on scikit-learn. This would be an awesome move! It'd also be much less disruptive, in case there are others using `label_encode` themselves (myself included).

We also could entirely deprecate `label_encode` and replace with `factorize_columns`. This would have the effect of nudging us closer to the educational effect of letting users understand that this is a convenience method that extends the pandas-native `factorize` function.

As I'm thinking about it, the `factorize_columns` function probably should have a keyword argument that specifies the desired suffix. Empty suffix means we replace the existing column, while new suffix means we create a new column with the appropriate suffix. `_enc` can remain the default keyword argument. We don't have to bring this new behaviour into the existing `label_encode` function since it's going to be deprecated.

Now that I've typed this out, I think the best course of action at this moment of time is as follows:

1. For an upcoming PR, swap out the `label_encode` function implementation to use `factorize` underneath. (Details will need to be worked out.)
2. In the same PR, add a warning that says that `label_encode` will be deprecated in a 1.x release so that until 1.x is released, everybody using `pyjanitor` is aware of the change.
3. In the same or in a future PR, add in the `factorize_columns` function with associated tests.

@nvamsikrishna05 what do you think about this plan? If it sounds good to you, I'd like to also invite you into the dev team after finishing this PR!
@ericmjl It's fine, don't worry about the delay. Thanks for the clarifications. It sounds good. I'll proceed with the implementations. Thank you 🙂
Once the Final PR for factorize_columns is merged,Will close this Issue.

Resolving patch:
diff --git a/janitor/functions.py b/janitor/functions.py
--- a/janitor/functions.py
+++ b/janitor/functions.py
@@ -777,11 +777,65 @@ def label_encode(
         or tuple) of column names.
     :returns: A pandas DataFrame.
     """
-    warnings.warn("label_encode will be deprecated in a 1.x release")
+    warnings.warn(
+        "label_encode will be deprecated in a 1.x release. \
+        Please use factorize_columns instead"
+    )
     df = _factorize(df, column_names, "_enc")
     return df
 
 
+@pf.register_dataframe_method
+def factorize_columns(
+    df: pd.DataFrame,
+    column_names: Union[str, Iterable[str], Hashable],
+    suffix: str = "_enc",
+    **kwargs,
+) -> pd.DataFrame:
+    """Converts labels into numerical data
+
+    This method will create a new column with the string "_enc" appended
+    after the original column's name.
+    This can be overriden with the suffix parameter
+
+    Internally this method uses pandas factorize method.
+    It takes in optional suffix and keyword arguments also.
+    An empty string as suffix will override the existing column
+
+    This method mutates the original DataFrame
+
+    Functional usage syntax:
+
+    .. code-block:: python
+
+        df = factorize_columns(df, column_names="my_categorical_column",
+                                        suffix="_enc")  # one way
+
+    Method chaining syntax:
+
+    .. code-block:: python
+
+        import pandas as pd
+        import janitor
+        categorical_cols = ['col1', 'col2', 'col4']
+        df = pd.DataFrame(...).factorize_columns(
+                                column_names=categorical_cols,
+                                suffix="_enc")
+
+    :param df: The pandas DataFrame object.
+    :param column_names: A column name or an iterable (list
+        or tuple) of column names.
+    :param suffix: Suffix to be used for the new column. Default value is _enc.
+        An empty string suffix means, it will override the existing column
+    :param **kwargs: Keyword arguments. It takes any of the keyword arguments,
+        which the pandas factorize method takes like sort,na_sentinel,size_hint
+
+    :returns: A pandas DataFrame.
+    """
+    df = _factorize(df, column_names, suffix, **kwargs)
+    return df
+
+
 @dispatch(pd.DataFrame, (list, tuple), str)
 def _factorize(df, column_names, suffix, **kwargs):
     check_column(df, column_names=column_names, present=True)


Test patch:
diff --git a/tests/functions/test_factorize_columns.py b/tests/functions/test_factorize_columns.py
new file mode 100644
--- /dev/null
+++ b/tests/functions/test_factorize_columns.py
@@ -0,0 +1,83 @@
+"""
+Author: Vamsi Krishna
+Date: 23 July 2021
+
+The intent of these tests is to test factorize_columns function works.
+Because underneath the hood we are using `pd.factorize`,
+we intentionally do not test the values of the resultant dataframe.
+That would be duplicating the tests from what the `pandas` library provides.
+"""
+import pandas as pd
+import pytest
+
+
+@pytest.mark.functions
+def test_single_column_factorize_columns():
+    """Tests if Single Column Factorize is working correctly"""
+    df = pd.DataFrame(
+        {"a": ["hello", "hello", "sup"], "b": [1, 2, 3]}
+    ).factorize_columns(column_names="a")
+    assert "a_enc" in df.columns
+
+
+@pytest.mark.functions
+def test_single_column_fail_factorize_columns():
+    """Tests if Single Column Factorize fails"""
+    with pytest.raises(ValueError):
+        pd.DataFrame(
+            {"a": ["hello", "hello", "sup"], "b": [1, 2, 3]}
+        ).factorize_columns(
+            column_names="c"
+        )  # noqa: 841
+
+
+@pytest.mark.functions
+def test_multicolumn_factorize_columns():
+    """Tests if Multi Column Factorize is working correctly"""
+    df = pd.DataFrame(
+        {
+            "a": ["hello", "hello", "sup"],
+            "b": [1, 2, 3],
+            "c": ["aloha", "nihao", "nihao"],
+        }
+    ).factorize_columns(column_names=["a", "c"])
+    assert "a_enc" in df.columns
+    assert "c_enc" in df.columns
+
+
+@pytest.mark.functions
+def test_multicolumn_factorize_columns_suffix_change():
+    """Tests if Multi Column Factorize works with suffix change"""
+    df = pd.DataFrame(
+        {
+            "a": ["hello", "hello", "sup"],
+            "b": [1, 2, 3],
+            "c": ["aloha", "nihao", "nihao"],
+        }
+    ).factorize_columns(column_names=["a", "c"], suffix="_col")
+    assert "a_col" in df.columns
+    assert "c_col" in df.columns
+    assert "a_enc" not in df.columns
+    assert "c_enc" not in df.columns
+
+
+@pytest.mark.functions
+def test_multicolumn_factorize_columns_empty_suffix():
+    """Tests if Multi Column Factorize works with empty suffix"""
+    df = pd.DataFrame(
+        {
+            "a": ["hello", "hello", "sup"],
+            "b": [1, 2, 3],
+            "c": ["aloha", "nihao", "nihao"],
+        }
+    ).factorize_columns(column_names=["a", "c"], suffix="")
+    assert "a_enc" not in df.columns
+    assert "c_enc" not in df.columns
+    assert 3 == len(df.columns)
+
+
+@pytest.mark.functions
+def test_factorize_columns_invalid_input(dataframe):
+    """Tests if Multi Column Factorize throws error"""
+    with pytest.raises(NotImplementedError):
+        dataframe.factorize_columns(1)
diff --git a/tests/test_df_registration.py b/tests/test_df_registration.py
--- a/tests/test_df_registration.py
+++ b/tests/test_df_registration.py
@@ -22,80 +22,105 @@ def dataframe():
 
 
 def test_clean_names_registration(dataframe):
+    """Test if DataFrame has clean_method method"""
     assert dataframe.__getattr__("clean_names")
 
 
 def test_remove_empty_registration(dataframe):
+    """Test if DataFrame has remove_empty method"""
     assert dataframe.__getattr__("remove_empty")
 
 
 def test_get_dupes_registration(dataframe):
+    """Test if DataFrame has get_dupes method"""
     assert dataframe.__getattr__("get_dupes")
 
 
 def test_encode_categorical_registration(dataframe):
+    """Test if DataFrame has encode_categorical method"""
     assert dataframe.__getattr__("encode_categorical")
 
 
 def test_label_encode_registration(dataframe):
+    """Test if DataFrame has label_encode method"""
     assert dataframe.__getattr__("label_encode")
 
 
+def test_factorize_columns_registration(dataframe):
+    """Test if DataFrame has factorize_columns method"""
+    assert dataframe.__getattr__("factorize_columns")
+
+
 def test_get_features_targets_registration(dataframe):
+    """Test if DataFrame has get_features_targets method"""
     assert dataframe.__getattr__("get_features_targets")
 
 
 def test_rename_column_registration(dataframe):
+    """Test if DataFrame has rename_column method"""
     assert dataframe.__getattr__("rename_column")
 
 
 def test_coalesce_registration(dataframe):
+    """Test if DataFrame has coalesce method"""
     assert dataframe.__getattr__("coalesce")
 
 
 def test_convert_excel_date_registration(dataframe):
+    """Test if DataFrame has convert_excel_date method"""
     assert dataframe.__getattr__("convert_excel_date")
 
 
 def test_convert_matlab_date_registration(dataframe):
+    """Test if DataFrame has convert_matlab_date method"""
     assert dataframe.__getattr__("convert_matlab_date")
 
 
 def test_convert_unix_date_registration(dataframe):
+    """Test if DataFrame has convert_unix_date method"""
     assert dataframe.__getattr__("convert_unix_date")
 
 
 def test_fill_empty_registration(dataframe):
+    """Test if DataFrame has fill_empty method"""
     assert dataframe.__getattr__("fill_empty")
 
 
 def test_expand_column_registration(dataframe):
+    """Test if DataFrame has expand_column method"""
     assert dataframe.__getattr__("expand_column")
 
 
 def test_concatenate_columns_registration(dataframe):
+    """Test if DataFrame has concatenate_columns method"""
     assert dataframe.__getattr__("concatenate_columns")
 
 
 def test_deconcatenate_column_registration(dataframe):
+    """Test if DataFrame has deconcatenate_column method"""
     assert dataframe.__getattr__("deconcatenate_column")
 
 
 def test_filter_string_registration(dataframe):
+    """Test if DataFrame has filter_string method"""
     assert dataframe.__getattr__("filter_string")
 
 
 def test_filter_on_registration(dataframe):
+    """Test if DataFrame has filter_on method"""
     assert dataframe.__getattr__("filter_on")
 
 
 def test_remove_columns_registration(dataframe):
+    """Test if DataFrame has remove_columns method"""
     assert dataframe.__getattr__("remove_columns")
 
 
 def test_change_type_registration(dataframe):
+    """Test if DataFrame has change_type method"""
     assert dataframe.__getattr__("change_type")
 
 
 def test_filter_date_registration(dataframe):
+    """Test if DataFrame has filter_date method"""
     assert dataframe.__getattr__("filter_date")