Repository: pyjanitor-devs/pyjanitor

Issue:
Deprecate label_encode?
I recently stumbled upon pandas' function `factorize`. I think it does what we want with `label_encode`, except it is not a dataframe method.

I'd like to propose the following course of action:

1. First off, switch out the label_encode implementation to use pandas' factorize.
2. Add an alias function `factorize` that enables this to be a dataframe method.
3. Add a warning that we will be deprecating `label_encode`.
4. Deprecate label_encode in the 1.x release.

What are everybody's thoughts on this one?


Discussion:
@ericmjl @samukweku Is this being worked on by anyone ? If not, I would like to take this up.  Thanks.
@nvamsikrishna05 pls go ahead.
@samukweku @ericmjl  Thanks

Just want to write down the thoughts here. Please correct me if I'm wrong.
(1)
The Current Solution - 
``` python
def label_encode(
    df: pd.DataFrame, column_names: Union[str, Iterable[str], Hashable]
) -> pd.DataFrame:
    df = _factorize(df, column_names)
    return df

@dispatch(pd.DataFrame, (list, tuple))
def _label_encode(df, column_names):
    le = LabelEncoder()
    check_column(df, column_names=column_names, present=True)
    for col in column_names:
        df[f"{col}_enc"] = le.fit_transform(df[col])
    return df

@dispatch(pd.DataFrame, str)  # noqa: F811
def _label_encode(df, column_names):  # noqa: F811
    le = LabelEncoder()
    check_column(df, column_names=column_names, present=True)
    df[f"{column_names}_enc"] = le.fit_transform(df[column_names])
    return df

```

Solution I'm thinking of with factorize would be - 

``` python

@dispatch(pd.DataFrame, (list, tuple))
def _factorize(df, column_names):
    check_column(df, column_names=column_names, present=True)
    for col in column_names:
        df[f"{col}_enc"] = pd.factorize(df[col])[0]
    return df

@dispatch(pd.DataFrame, str)
def _factorize(df, column_name):
    check_column(df, column_names=column_name, present=True)
    df[f"{column_name}_enc"] = pd.factorize(df[column_name])[0]
    return df
```
These two would be replacing the _label_encode implementations.

The main benefit of doing this would be we would be wrapping over the pandas implementation and can remove the sklearn LabelEncoder. I see that sklearn is directly used just for this particular function only. so we can remove it ?

(3) For adding a warning that label_encode is this the right approach or are we using something else ?
``` python
 warnings.warn("label_encode will be deprecated in future release. Please use factorize", DeprecationWarning)
```
This would be at the start of label_encode

(2) Add an alias factorize -  another function like
``` python
def factorize(df, column_name):
    pass
```
In this way like a function overloading over the pandas implementation ? Wouldn't that conflict over pandas one ?

Couple of questions I have -
The pandas factorize implementation has optional parameter -
 na_sentinel=- 1 # used to convert NaN. Default is -1 which can be changed.
sort = False  # used to sort the unique value and then assign the numbering.
Should we be providing this option ? LabelEncoder doesn't have it so we didn't have it so far.

Hi @nvamsikrishna05! Thanks for thinking through this issue so thoroughly. I am going to answer your questions one by one.

> I see that sklearn is directly used just for this particular function only. so we can remove it ?

Yes! That is correct; it's always nice to minimize the number of dependencies of a package, especially big ones. 

> For adding a warning that label_encode is this the right approach or are we using something else ?

The implementation you suggested looks about correct.

> In this way like a function overloading over the pandas implementation ? Wouldn't that conflict over pandas one ?

You are also correct here. What do you think of providing a `factorize_columns` function instead? I'm thinking of this because one value add `pyjanitor` provides over `pandas` is to make convenient some pandas operations, i.e. 

> The pandas factorize implementation has optional parameter -
> na_sentinel=- 1 # used to convert NaN. Default is -1 which can be changed.
> sort = False # used to sort the unique value and then assign the numbering.
> Should we be providing this option ? LabelEncoder doesn't have it so we didn't have it so far.

It makes sense to pass-through keyword arguments from `factorize_columns` into `factorize`. I think it's sensible to start simple first and assume that `factorize` keyword arguments are applied uniformly across all columns on which we are applying factorize, unless there's a better pattern that we can follow.

The pattern here would look akin to:

```python
def factorize_columns(df, column_names, **factorize_kwargs):
    """Docstring goes here."""
    for column in column_names:
        df[f"{column}_enc"] = pd.factorize(df[column], **factorize_kwargs)
    return df
```
@ericmjl Thanks for the clarifications.  Yes, factorize_columns name sounds good. Also should the keyword arguments be added to the to-be deprecated label_encode function also ?

About retiring the label_encode function, doesn't "label_encode" name is more meaningful for people looking to do Label Encoding quickly ? Is the reasoning behind using factorize_columns so that people understand that this is the convenience method over the pandas factorize (mind map linking 1-1) ?
@nvamsikrishna05 you raise a great question there! Thanks for your patience, btw, I was catching up with life after doing my SciPy tutorial on Tuesday. I also took the time to ponder over your question on-and-off.

Indeed, because `pyjanitor` is really about convenience functions, I think there are two ways we could proceed here. We could replace just the `label_encode` implementation, with the effect of removing `pyjanitor`'s dependency on scikit-learn. This would be an awesome move! It'd also be much less disruptive, in case there are others using `label_encode` themselves (myself included).

We also could entirely deprecate `label_encode` and replace with `factorize_columns`. This would have the effect of nudging us closer to the educational effect of letting users understand that this is a convenience method that extends the pandas-native `factorize` function.

As I'm thinking about it, the `factorize_columns` function probably should have a keyword argument that specifies the desired suffix. Empty suffix means we replace the existing column, while new suffix means we create a new column with the appropriate suffix. `_enc` can remain the default keyword argument. We don't have to bring this new behaviour into the existing `label_encode` function since it's going to be deprecated.

Now that I've typed this out, I think the best course of action at this moment of time is as follows:

1. For an upcoming PR, swap out the `label_encode` function implementation to use `factorize` underneath. (Details will need to be worked out.)
2. In the same PR, add a warning that says that `label_encode` will be deprecated in a 1.x release so that until 1.x is released, everybody using `pyjanitor` is aware of the change.
3. In the same or in a future PR, add in the `factorize_columns` function with associated tests.

@nvamsikrishna05 what do you think about this plan? If it sounds good to you, I'd like to also invite you into the dev team after finishing this PR!
@ericmjl It's fine, don't worry about the delay. Thanks for the clarifications. It sounds good. I'll proceed with the implementations. Thank you 🙂

Resolving patch:
diff --git a/janitor/functions.py b/janitor/functions.py
--- a/janitor/functions.py
+++ b/janitor/functions.py
@@ -31,7 +31,6 @@
 from pandas.api.types import is_bool_dtype, is_list_like, union_categoricals
 from pandas.errors import OutOfBoundsDatetime
 from scipy.stats import mode
-from sklearn.preprocessing import LabelEncoder
 
 from .errors import JanitorError
 from .utils import (
@@ -778,24 +777,23 @@ def label_encode(
         or tuple) of column names.
     :returns: A pandas DataFrame.
     """
-    df = _label_encode(df, column_names)
+    warnings.warn("label_encode will be deprecated in a 1.x release")
+    df = _factorize(df, column_names, "_enc")
     return df
 
 
-@dispatch(pd.DataFrame, (list, tuple))
-def _label_encode(df, column_names):
-    le = LabelEncoder()
+@dispatch(pd.DataFrame, (list, tuple), str)
+def _factorize(df, column_names, suffix, **kwargs):
     check_column(df, column_names=column_names, present=True)
     for col in column_names:
-        df[f"{col}_enc"] = le.fit_transform(df[col])
+        df[f"{col}{suffix}"] = pd.factorize(df[col], **kwargs)[0]
     return df
 
 
-@dispatch(pd.DataFrame, str)  # noqa: F811
-def _label_encode(df, column_names):  # noqa: F811
-    le = LabelEncoder()
-    check_column(df, column_names=column_names, present=True)
-    df[f"{column_names}_enc"] = le.fit_transform(df[column_names])
+@dispatch(pd.DataFrame, str, str)
+def _factorize(df, column_name, suffix, **kwargs):  # noqa: F811
+    check_column(df, column_names=column_name, present=True)
+    df[f"{column_name}{suffix}"] = pd.factorize(df[column_name], **kwargs)[0]
     return df
 
 


Test patch: