Repository: scikit-learn/scikit-learn
Pull request: ENH Add 'if_binary' option to drop argument of OneHotEncoder
Fixes #12502

Problem statement:
RFC Behavior of OneHotEncoder on binary features
Right now OneHotEncoder expands every bindary feature into two features afaik.
I'm not sure if this is a great/convenient behavior. I think it might be nicer if (optionally?) it'd use a single column - that's particularly natural if the feature was already ``0`` and ``1``
Otherwise that either makes the ColumnTransformer people have to use more complicated, or creates redundant features.
One might argue that one possible cure for that is the option to drop one of the indicator variables. I'm not really sure if that's what I want, though. In my mind having a base category is more interpretable in the binary case than in the multinomial case.

Hints from issue discussion:
I agree. Using a single feature in case of binary makes sense. In the multinomial case, as far as I have read, models like linear regression and GLM, which uses maximum likelihood will have issues due to singularities. There are LeaveOneOut encoders for handling such model requirements. I think, keeping just one feature for binary features would be a good idea here.  
I think that we should drop one of the columns by default.
It would be great to have this! 

A way forward could be to add a e.g. `reduce_bool=False` (or some other name) parameter and then discuss whether we should add a deprecation cycle to make it `True` by default.

A quick and hacky implementation in case someone wants to play with this could be,
```py
class UniqueBinaryOneHotEncoder(OneHotEncoder):
    """OneHotEncoder that doesn't create redundant binary features
    
    See sklearn.preprocessing.OneHotEncoder for more details
    
    Examples
    --------
    >>> X = [['blue', 0], ['green', 1], ['orange', 0]]
    >>> enc = UniqueBinaryOneHotEncoder(sparse=False)
    >>> enc.fit_transform(X)
    array([[1., 0., 0., 0.],
           [0., 1., 0., 1.],
           [0., 0., 1., 0.]])
    >>> enc.get_feature_names()
    array(['x0_blue', 'x0_green', 'x0_orange', 'x1_1'], dtype=object)
    """
    def __init__(self, sparse=False):
        super().__init__(sparse=sparse)
    
    def _get_binary_column_mask(self):
        n_features = sum(len(cat) for cat in self.categories_)
        idx = 0
        mask = np.ones(n_features, dtype=np.bool)
        for cat in self.categories_:
            if len(cat) == 2:
                # binary feature, drop the first column
                mask[idx] = 0
            idx += len(cat)
        return mask

    def transform(self, X):
        Y = super().transform(X)
        
        mask = self._get_binary_column_mask()
            
        return Y[:, mask]
    
    def inverse_transform(self, X):
        raise NotImplementedError()
    
    def get_feature_names(self, input_features=None):
        mask = self._get_binary_column_mask()
        return super().get_feature_names(input_features)[mask]
```
(I'm not saying that there isn't a better approach for it when making the actual PR).


The PR to add a `drop` keyword to OneHotEncoder (https://github.com/scikit-learn/scikit-learn/pull/12908) is making good progress and is almost ready to be merged.

How do we see this issue in relation to the `drop` keyword? 
Does it solve it? Or do we still want to have a way to only drop one feature in the binary case? (eg we could add a `drop='if_binary'` in addition to `drop='first'`) 
Or do we actually want to change the default?


cc @drewmjohnston @NicolasHug @jnothman 
I'd be happy to add drop='binary' and eventually change the default. But
I'm okay with having drop='first' merged as a first go.

Yes, sure, it doesn't need to be added to the open PR (but it would be good if a way forward for this issue is compatible with what we are doing there)
>I'd be happy to add drop='binary' and eventually change the default. But I'm okay with having drop='first' merged as a first go.

+1
I've been curious if anyone has an understanding of the theory and/or established practice behind dropping one of the K levels to make K-1 columns, versus keeping all of them. I include my brief understanding below.

### In our tools
It seems that `sklearn.OneHotEncoder` has, for a long time, kept all K levels and it seems that the community has been fine with it. I believe `DictVectorizer` also keeps all K levels during transform.

Similarly, pandas `pd.get_dummies()`,  until version 0.18, kept all K levels, and still keeps them by default, but you can now add `drop_first=True` to remove one level.

### In statistics
As @aishgrt1 states above, in statistics, it is common to drop one level to prevent creating singular matrices and causing multicollinearity. I believe this is because traditional statistical models deal with relatively "small" data with not many columns (I am not sure though), and singular matrices can be a problem.

### In machine learning
However, my hunch is that this matters less in cases where you have lots of records and/or many columns, and that is why, traditionally, ML folks didn't seem to care about keeping all K levels (as evidenced by the defaults in the tools listed above).

---

What do others think? (Not sure if this is the right place to include this discussion)
@pavopax there are a couple of other threads in which people discuss some useful information about this--dropping a value seems to matter most in regression (both unregularized and regularized). In the second case, it effects the value of the coefficients, which is important for interpretation. In the second case, this can even effect which columns drop out, which can lead to different solutions. Perfect collinearity also plays poorly with some other models, such as Keras neural networks.

Commit message: add isinstance of str condition

Patch to review:
diff --git a/doc/whats_new/v0.23.rst b/doc/whats_new/v0.23.rst
index 1941aacb7a7b0..79e521538d8c7 100644
--- a/doc/whats_new/v0.23.rst
+++ b/doc/whats_new/v0.23.rst
@@ -114,6 +114,10 @@ Changelog
 
 - |Efficiency| :class:`preprocessing.OneHotEncoder` is now faster at
   transforming. :pr:`15762` by `Thomas Fan`_.
+- |Feature| argument `drop` of :class:`preprocessing.OneHotEncoder`
+  will now accept value 'if_binary' and will drop the first category of
+  each binary features for `drop = 'if_binary'`. :pr:`#16245`
+  by :user:`Rushabh Vasani <rushabh-v>`.
 
 :mod:`sklearn.tree`
 ...................
diff --git a/sklearn/preprocessing/_encoders.py b/sklearn/preprocessing/_encoders.py
index 36512e359c7ed..6e7c694d02924 100644
--- a/sklearn/preprocessing/_encoders.py
+++ b/sklearn/preprocessing/_encoders.py
@@ -195,6 +195,7 @@ class OneHotEncoder(_BaseEncoder):
         - None : retain all features (the default).
         - 'first' : drop the first category in each feature. If only one
           category is present, the feature will be dropped entirely.
+        - 'if_binary' : drop the first category in each binary feature.
         - array : ``drop[i]`` is the category in feature ``X[:, i]`` that
           should be dropped.
 
@@ -293,7 +294,9 @@ def _validate_keywords(self):
     def _compute_drop_idx(self):
         if self.drop is None:
             return None
-        elif (isinstance(self.drop, str) and self.drop == 'first'):
+        elif (isinstance(self.drop, str)
+              and (self.drop == 'first'
+              or self.drop == 'if_binary')):
             return np.zeros(len(self.categories_), dtype=np.int_)
         elif not isinstance(self.drop, str):
             try:
@@ -323,7 +326,7 @@ def _compute_drop_idx(self):
                              zip(self.drop, self.categories_)], dtype=np.int_)
         else:
             msg = ("Wrong input for parameter `drop`. Expected "
-                   "'first', None or array of objects, got {}")
+                   "'first', 'if_binary', None or array of objects, got {}")
             raise ValueError(msg.format(type(self.drop)))
 
     def fit(self, X, y=None):
@@ -398,9 +401,25 @@ def transform(self, X):
             # categories that occur after them to avoid an empty column.
 
             keep_cells = X_int != to_drop
+
+            if isinstance(self.drop, str) and self.drop == 'if_binary':
+                keep_cells = keep_cells.T
+                X_int = X_int.T
+                for i in range(n_features):
+                    if len(self.categories_[i]) > 2:
+                        keep_cells[i] = True
+                    else:
+                        X_int[i][X_int[i] > 0] -= 1
+                keep_cells = keep_cells.T
+                X_int = X_int.T
+                n_values = [1 if len(cats) == 2
+                            else len(cats) for cats in self.categories_]
+
+            else:
+                X_int[X_int > to_drop] -= 1
+                n_values = [len(cats) - 1 for cats in self.categories_]
+
             X_mask &= keep_cells
-            X_int[X_int > to_drop] -= 1
-            n_values = [len(cats) - 1 for cats in self.categories_]
         else:
             n_values = [len(cats) for cats in self.categories_]
 
diff --git a/sklearn/preprocessing/tests/test_encoders.py b/sklearn/preprocessing/tests/test_encoders.py
index 78590f40ffba5..f2dec3f767156 100644
--- a/sklearn/preprocessing/tests/test_encoders.py
+++ b/sklearn/preprocessing/tests/test_encoders.py
@@ -398,6 +398,15 @@ def test_one_hot_encoder_feature_names_drop(drop, expected_names):
     assert_array_equal(expected_names, feature_names)
 
 
+def test_one_hot_encoder_drop_equals_if_binary():
+    X = [['Male', 1], ['Female', 3], ['Female', 2]]
+    expected = np.array([[1., 1., 0., 0.], [0., 0., 0., 1.], [0., 0., 1., 0.]])
+    ohe = OneHotEncoder(drop='if_binary')
+    ohe.fit(X)
+    result = ohe.transform(X).toarray()
+    assert_array_equal(expected, result)
+
+
 @pytest.mark.parametrize("X", [np.array([[1, np.nan]]).T,
                                np.array([['a', np.nan]], dtype=object).T],
                          ids=['numeric', 'object'])


Code review comments:
diff_hunk: @@ -398,6 +398,15 @@ def test_one_hot_encoder_feature_names_drop(drop, expected_names):
     assert_array_equal(expected_names, feature_names)
 
 
+def test_one_hot_encoder_drop_equals_if_binary():
+    X = [['Male', 1], ['Female', 3], ['Female', 2]]
+    expected = np.array([[1., 1., 0., 0.], [0., 0., 0., 1.], [0., 0., 1., 0.]])
+    ohe = OneHotEncoder(drop='if_binary')
+    ohe.fit(X)
+    result = ohe.transform(X).toarray()
+    assert_array_equal(expected, result)
path: sklearn/preprocessing/tests/test_encoders.py
text: @user1:
```suggestion
    assert_allclose(expected, result)
```
The array is of dtype float

diff_hunk: @@ -293,7 +294,9 @@ def _validate_keywords(self):
     def _compute_drop_idx(self):
         if self.drop is None:
             return None
-        elif (isinstance(self.drop, str) and self.drop == 'first'):
+        elif (isinstance(self.drop, str)
path: sklearn/preprocessing/_encoders.py
text: @user1:
```suggestion
        elif isinstance(self.drop, str) and self.drop in ['first', 'if_binary']:
```

diff_hunk: @@ -398,6 +398,15 @@ def test_one_hot_encoder_feature_names_drop(drop, expected_names):
     assert_array_equal(expected_names, feature_names)
 
 
+def test_one_hot_encoder_drop_equals_if_binary():
+    X = [['Male', 1], ['Female', 3], ['Female', 2]]
+    expected = np.array([[1., 1., 0., 0.], [0., 0., 0., 1.], [0., 0., 1., 0.]])
+    ohe = OneHotEncoder(drop='if_binary')
+    ohe.fit(X)
path: sklearn/preprocessing/tests/test_encoders.py
text: @user1:
remove

diff_hunk: @@ -398,9 +401,25 @@ def transform(self, X):
             # categories that occur after them to avoid an empty column.
 
             keep_cells = X_int != to_drop
+
+            if isinstance(self.drop, str) and self.drop == 'if_binary':
+                keep_cells = keep_cells.T
path: sklearn/preprocessing/_encoders.py
text: @user1:
Why do you transpose here?

@user1:
We could just iterate the same way directly on the column?

diff_hunk: @@ -114,6 +114,10 @@ Changelog
 
 - |Efficiency| :class:`preprocessing.OneHotEncoder` is now faster at
   transforming. :pr:`15762` by `Thomas Fan`_.
+- |Feature| argument `drop` of :class:`preprocessing.OneHotEncoder`
+  will now accept value 'if_binary' and will drop the first category of
+  each binary features for `drop = 'if_binary'`. :pr:`#16245`
path: doc/whats_new/v0.23.rst
text: @user1:
```suggestion
  each binary features for `drop='if_binary'`. :pr:`#16245`
```

diff_hunk: @@ -398,9 +401,25 @@ def transform(self, X):
             # categories that occur after them to avoid an empty column.
 
             keep_cells = X_int != to_drop
+
+            if isinstance(self.drop, str) and self.drop == 'if_binary':
+                keep_cells = keep_cells.T
+                X_int = X_int.T
+                for i in range(n_features):
+                    if len(self.categories_[i]) > 2:
+                        keep_cells[i] = True
path: sklearn/preprocessing/_encoders.py
text: @user1:
```suggestion
                        keep_cells[:, col] = True
```

diff_hunk: @@ -398,9 +401,25 @@ def transform(self, X):
             # categories that occur after them to avoid an empty column.
 
             keep_cells = X_int != to_drop
+
+            if isinstance(self.drop, str) and self.drop == 'if_binary':
+                keep_cells = keep_cells.T
+                X_int = X_int.T
+                for i in range(n_features):
+                    if len(self.categories_[i]) > 2:
path: sklearn/preprocessing/_encoders.py
text: @user1:
```suggestion
                    if len(self.categories_[col]) > 2:
```

diff_hunk: @@ -398,9 +401,25 @@ def transform(self, X):
             # categories that occur after them to avoid an empty column.
 
             keep_cells = X_int != to_drop
+
+            if isinstance(self.drop, str) and self.drop == 'if_binary':
+                keep_cells = keep_cells.T
+                X_int = X_int.T
+                for i in range(n_features):
path: sklearn/preprocessing/_encoders.py
text: @user1:
```suggestion
                for col in range(n_features):
```

diff_hunk: @@ -398,9 +401,25 @@ def transform(self, X):
             # categories that occur after them to avoid an empty column.
 
             keep_cells = X_int != to_drop
+
+            if isinstance(self.drop, str) and self.drop == 'if_binary':
+                keep_cells = keep_cells.T
path: sklearn/preprocessing/_encoders.py
text: @user1:
I think you can avoid these transpositions.

diff_hunk: @@ -398,6 +398,15 @@ def test_one_hot_encoder_feature_names_drop(drop, expected_names):
     assert_array_equal(expected_names, feature_names)
 
 
+def test_one_hot_encoder_drop_equals_if_binary():
+    X = [['Male', 1], ['Female', 3], ['Female', 2]]
+    expected = np.array([[1., 1., 0., 0.], [0., 0., 0., 1.], [0., 0., 1., 0.]])
+    ohe = OneHotEncoder(drop='if_binary')
+    ohe.fit(X)
+    result = ohe.transform(X).toarray()
path: sklearn/preprocessing/tests/test_encoders.py
text: @user1:
```suggestion
    X_trans = ohe.fit_transform(X)
```

diff_hunk: @@ -398,9 +401,25 @@ def transform(self, X):
             # categories that occur after them to avoid an empty column.
 
             keep_cells = X_int != to_drop
+
+            if isinstance(self.drop, str) and self.drop == 'if_binary':
+                keep_cells = keep_cells.T
+                X_int = X_int.T
+                for i in range(n_features):
+                    if len(self.categories_[i]) > 2:
+                        keep_cells[i] = True
+                    else:
+                        X_int[i][X_int[i] > 0] -= 1
+                keep_cells = keep_cells.T
+                X_int = X_int.T
path: sklearn/preprocessing/_encoders.py
text: @user1:
No need to transpose

diff_hunk: @@ -398,9 +401,25 @@ def transform(self, X):
             # categories that occur after them to avoid an empty column.
 
             keep_cells = X_int != to_drop
+
+            if isinstance(self.drop, str) and self.drop == 'if_binary':
+                keep_cells = keep_cells.T
+                X_int = X_int.T
+                for i in range(n_features):
+                    if len(self.categories_[i]) > 2:
+                        keep_cells[i] = True
path: sklearn/preprocessing/_encoders.py
text: @user1:
If keep_cells is not transposed, you should be able to use: `keep_cells[:, i] = True`

diff_hunk: @@ -398,6 +398,15 @@ def test_one_hot_encoder_feature_names_drop(drop, expected_names):
     assert_array_equal(expected_names, feature_names)
 
 
+def test_one_hot_encoder_drop_equals_if_binary():
+    X = [['Male', 1], ['Female', 3], ['Female', 2]]
+    expected = np.array([[1., 1., 0., 0.], [0., 0., 0., 1.], [0., 0., 1., 0.]])
+    ohe = OneHotEncoder(drop='if_binary')
path: sklearn/preprocessing/tests/test_encoders.py
text: @user1:
```suggestion
    ohe = OneHotEncoder(drop='if_binary', sparse=False)
```

diff_hunk: @@ -398,9 +401,25 @@ def transform(self, X):
             # categories that occur after them to avoid an empty column.
 
             keep_cells = X_int != to_drop
+
+            if isinstance(self.drop, str) and self.drop == 'if_binary':
+                keep_cells = keep_cells.T
+                X_int = X_int.T
+                for i in range(n_features):
+                    if len(self.categories_[i]) > 2:
+                        keep_cells[i] = True
+                    else:
+                        X_int[i][X_int[i] > 0] -= 1
path: sklearn/preprocessing/_encoders.py
text: @user1:
If X_int was not transposed, `X_int[X_int[i] > 0, i] -= 1` should work.

@author:
That raises this error `boolean index did not match indexed array along dimension 0; dimension is 3 but corresponding boolean dimension is 5`.
Using `X_int[:, col][X_int[:, col] > 0] -= 1` instead.

@user2:
you need to put the masking in the first indexing `[..., ...]`