Repository: scikit-learn/scikit-learn
Pull request: [MRG + 2] Add Drop Option to OneHotEncoder.
#### Reference Issues/PRs

Fixes #6488 and fixes #6053. This builds upon some of the code from #12884 (thanks @NicolasHug!), but also incorporates functionality which lets the user manually specify which category in each column they would like to be dropped, so this is a more general solution along the lines of what @amueller suggested in #6053. This is useful in some cases (such as OLS regression) where the dropped group affects the interpretation of coefficients. 
This should also fix #9361, which has less functionality and appears stalled.

#### What does this implement/fix? Explain your changes.
This code implements a new parameter (drop) in the OneHotEncoder, which can take any of three values: 
None (default), which implements the existing behavior. 
'first', which drops the first value in each category.
list of length n_features, which allows for the manual specification of the reference group. 

#### Any other comments?

This new feature does not work in Legacy mode (this was discussed in #6053), and it requires the manual specification of "categories='auto'" in the case in which the input is all integers, so as not to interfere with the ongoing change in the treatment of integers in OneHotEncoder.
This code is also incompatible with "handle_missing='ignore'", since in that case it is not possible to determine which categories are 0 because they are in the reference category and which are all 0 due to missing data.

Closes #12884

Problem statement:
OneHotEncoding - Defining a reference category 
In order to avoid multicollinearity in modelling, the number of dummy-coded variables needed should be one less than the number of categories. Therefore, it would be very code if OneHotEncoding could accept a reference category as an input variable. 


OneHotEncoder - add option for 1 of k-1 encoding
Like the title says. Would it be possible to add an option, say "independent = True" to OneHotEncoder that would return a 1 of k-1 encoding instead of a 1 of k encoding. This would be very useful to me when I am encoding categorical variables since the 1 of k encoding adds an extra (non-independent) degree of freedom to the model. It would also be nice if I could specify which category to keep as the baseline.

Something like:

```
X = np.array([12,24,36]).reshape(-1,1)  
OneHotEncoder(sparse=False, independent=True, baseline=24).fit_transform(X)  
Output: array([[ 1., 0.],
       [ 0., 0.],
       [ 0., 1.]])
```

Hints from issue discussion:
not sure if that was raise somewhere already. multicollinearity is really not a problem in any model in scikit-learn. But feel free to create a pull-request. The OneHotEncoder is being restructured quite heavily right now, though.

I'm interested in working on this feature! I ran into some problems using a OneHotEncoder in a pipeline that used a Keras Neural Network as the classifier. I was attempting to transform a few columns of categorical features into a dummy variable representation and feed the resulting columns (plus some numerical variables that were passed through) into the NN for classification. However, the one hot encoding played poorly with the collinear columns, and my model performed poorly out of sample. I was eventually able to design a workaround, but it seems to me that it would be valuable to have a tool in scikit-learn that could do this simply. 
I see the above pull request, which began to implement this in the DictVectorizer class, but it looks like this was never implemented (probably due to some unresolved fixes that were suggested). Is there anything stopping this from being implemented in the OneHotEncoder case instead?
I think we'd accept a PR. I'm a bit surprised there's none yet. We also changed the OneHotEncoder quite a bit recently. You probably don't want to modify the "legacy" mode. A question is whether/how we allow users to specify which category to drop. In regularized models this actually makes a difference IIRC.
We could have a parameter ``drop`` that's ``'none'`` by default, and could be ``'first'`` or a datastructure with the values to drop. could be a list/numpy array of length n_features (all input features are categorical in the new OneHotEncoder).
Reading through the comments on the old PR, I was thinking that those
options seem to be the natural choice. I'm in the midst of graduate school
applications right now so my time is somewhat limited, but this seems to be
something that is going to keep appearing in my work, so I'm going to have
to address this (or keep using workarounds) at some point.

On Wed, Nov 28, 2018 at 3:49 PM Andreas Mueller <notifications@github.com>
wrote:

> I think we'd accept a PR. I'm a bit surprised there's none yet. We also
> changed the OneHotEncoder quite a bit recently. You probably don't want to
> modify the "legacy" mode. A question is whether/how we allow users to
> specify which category to drop. In regularized models this actually makes a
> difference IIRC.
> We could have a parameter drop that's 'none' by default, and could be
> 'first' or a datastructure with the values to drop. could be a list/numpy
> array of length n_features (all input features are categorical in the new
> OneHotEncoder).
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/scikit-learn/scikit-learn/issues/6053#issuecomment-442599181>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/Am4ucQaSxhMTVGCeWx4cy-xx5Xl3EHmOks5uzvbugaJpZM4G2bMF>
> .
>

I've got a working implementation of this now--going to draw up some tests and then submit a PR later this week.
@drewmjohnston I think https://github.com/scikit-learn/scikit-learn/pull/12884 solves this issue right?
Yes, I think it should! I've also added functionality for manually specifying the dropped value for each feature.
(this is along the lines of what @amueller suggested)
Ok.

Please add a reference to #12884 in your PR so reviewers know that both PRs address the same thing, yours being more general. Also feel free to use some of my code if you deem relevant.
Sounds great--I think your testing suite will come in handy. Thank you for
the help :).

On Wed, Jan 2, 2019 at 2:53 PM Nicolas Hug <notifications@github.com> wrote:

> Ok.
>
> Please add a reference to #12884
> <https://github.com/scikit-learn/scikit-learn/pull/12884> in your PR so
> reviewers know that both PRs address the same thing, yours being more
> general. Also feel free to use some of my code if you deem relevant.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/scikit-learn/scikit-learn/issues/6053#issuecomment-450966515>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/Am4ucSK38aUyJ14bEUMYjXXgUUhWFJX9ks5u_Q5SgaJpZM4G2bMF>
> .
>

I guess we could do that as many people ask about it. I don't think there is that much of a point in doing that. Nearly all the models in scikit-learn are regularized, so this doesn't matter afaik.
I guess you are using a linear model?

@vighneshbirodkar is working on the `OneHotEncoder`, it might be worth waiting till that is finished.

Yup. I'm using a linear regression on categorical variables. So actually no regularization. 

The regression works fine so there must be a fix for collinearity (non invertibility) in `scipy.linalg.lstsq` but I find this way of building a model a bit confusing. The solution to the least squared problem with collinearity is under determined - there is a family of solutions. And so to solve it there is some behind the scenes choice being made as to _the_ solution which is hidden from the user. Basically I'd rather not introduce collinearity into a model that will then have to deal with that collinearity. 

This is all obviously just my under informed opinion :) 

Why are you not using regularization? I think the main reason people don't use regularization is that they want simple statistics on the coefficients. But scikit-learn doesn't provide any statistics of the coefficients. Maybe statsmodels would be a better fit for your application.

If you are interested in predictive performance, just replace `LinearRegression` with `RidgeCV` and your predictions will improve.

Ok. I guess regularization is the way to go in scikit-learn. I do still disagree in that I think dependence shouldn't be introduced into the model by way of preprocessors (or at least there should be an option to turn this off). But maybe this is getting at the difference between machine learning and statistical modelling. Or maybe who cares about independence if we have regularization.

Sorry for the slow reply. I'm ok with adding an option to turn this off. But as I said, OneHotEncoder is still being refactored.

I've elsewhere discussed similar for `LabelBinarizer` in the multiclass case, proposing the parameter name `drop_first` to ignore the encoding of the smallest value.

Commit message: Removed unnecessary loop on transform step

Patch to review:
diff --git a/doc/modules/preprocessing.rst b/doc/modules/preprocessing.rst
index 54210f2453cb0..3689ef6e95734 100644
--- a/doc/modules/preprocessing.rst
+++ b/doc/modules/preprocessing.rst
@@ -489,7 +489,7 @@ Continuing the example above::
   >>> enc = preprocessing.OneHotEncoder()
   >>> X = [['male', 'from US', 'uses Safari'], ['female', 'from Europe', 'uses Firefox']]
   >>> enc.fit(X)  # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
-  OneHotEncoder(categorical_features=None, categories=None,
+  OneHotEncoder(categorical_features=None, categories=None, drop=None,
          dtype=<... 'numpy.float64'>, handle_unknown='error',
          n_values=None, sparse=True)
   >>> enc.transform([['female', 'from US', 'uses Safari'],
@@ -516,7 +516,7 @@ dataset::
     >>> X = [['male', 'from US', 'uses Safari'], ['female', 'from Europe', 'uses Firefox']]
     >>> enc.fit(X) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
     OneHotEncoder(categorical_features=None,
-           categories=[...],
+           categories=[...], drop=None,
            dtype=<... 'numpy.float64'>, handle_unknown='error',
            n_values=None, sparse=True)
     >>> enc.transform([['female', 'from Asia', 'uses Chrome']]).toarray()
@@ -533,7 +533,7 @@ columns for this feature will be all zeros
     >>> enc = preprocessing.OneHotEncoder(handle_unknown='ignore')
     >>> X = [['male', 'from US', 'uses Safari'], ['female', 'from Europe', 'uses Firefox']]
     >>> enc.fit(X) # doctest: +ELLIPSIS  +NORMALIZE_WHITESPACE
-    OneHotEncoder(categorical_features=None, categories=None,
+    OneHotEncoder(categorical_features=None, categories=None, drop=None,
            dtype=<... 'numpy.float64'>, handle_unknown='ignore',
            n_values=None, sparse=True)
     >>> enc.transform([['female', 'from Asia', 'uses Chrome']]).toarray()
diff --git a/sklearn/preprocessing/_encoders.py b/sklearn/preprocessing/_encoders.py
index a03da8c653d06..6adfd311a4a46 100644
--- a/sklearn/preprocessing/_encoders.py
+++ b/sklearn/preprocessing/_encoders.py
@@ -2,11 +2,13 @@
 #          Joris Van den Bossche <jorisvandenbossche@gmail.com>
 # License: BSD 3 clause
 
+from __future__ import division
 
 import numbers
 import warnings
 
 import numpy as np
+import six
 from scipy import sparse
 
 from .. import get_config as _get_config
@@ -158,6 +160,21 @@ class OneHotEncoder(_BaseEncoder):
 
         The used categories can be found in the ``categories_`` attribute.
 
+    drop : 'first' or a list/array of shape (n_features,), default=None.
+        Specifies a methodology to use to drop one of the categories per
+        feature. This is useful in situations where perfectly collinear
+        features cause problems, such as when feeding the resulting data
+        into a neural network or an unregularized regression. If only one
+        feature appears in a column, that column will be dropped from the
+        transformed version (unless this is overriden by manually
+        specifying that no category ought to be dropped from that column).
+
+        - None : retain all features.
+        - 'first' : drop the first category in each feature.
+        - array : ``drop[i]`` is the value of the feature in ``X[:,i]``
+                  that should be dropped. None means all categories should
+                  be retained.
+
     sparse : boolean, default=True
         Will return sparse matrix if set True else will return an array.
 
@@ -207,6 +224,10 @@ class OneHotEncoder(_BaseEncoder):
         (in order of the features in X and corresponding with the output
         of ``transform``).
 
+    drop_idx_ : array of shape (n_features,)
+        The index in ``categories_`` of the category to be dropped for
+        each feature. None if all features will be retained.
+
     active_features_ : array
         Indices for active features, meaning values that actually occur
         in the training set. Only available when n_values is ``'auto'``.
@@ -243,9 +264,9 @@ class OneHotEncoder(_BaseEncoder):
     >>> enc.fit(X)
     ... # doctest: +ELLIPSIS
     ... # doctest: +NORMALIZE_WHITESPACE
-    OneHotEncoder(categorical_features=None, categories=None,
-           dtype=<... 'numpy.float64'>, handle_unknown='ignore',
-           n_values=None, sparse=True)
+    OneHotEncoder(categorical_features=None, categories=None, drop=None,
+       dtype=<... 'numpy.float64'>, handle_unknown='ignore',
+       n_values=None, sparse=True)
 
     >>> enc.categories_
     [array(['Female', 'Male'], dtype=object), array([1, 2, 3], dtype=object)]
@@ -257,6 +278,12 @@ class OneHotEncoder(_BaseEncoder):
            [None, 2]], dtype=object)
     >>> enc.get_feature_names()
     array(['x0_Female', 'x0_Male', 'x1_1', 'x1_2', 'x1_3'], dtype=object)
+    >>> drop_enc = OneHotEncoder(drop='first').fit(X)
+    >>> drop_enc.categories_
+    [array(['Female', 'Male'], dtype=object), array([1, 2, 3], dtype=object)]
+    >>> drop_enc.transform([['Female', 1], ['Male', 2]]).toarray()
+    array([[0., 0., 0.],
+           [1., 1., 0.]])
 
     See also
     --------
@@ -273,7 +300,7 @@ class OneHotEncoder(_BaseEncoder):
       matrix indicating the presence of a class label.
     """
 
-    def __init__(self, n_values=None, categorical_features=None,
+    def __init__(self, n_values=None, drop=None, categorical_features=None,
                  categories=None, sparse=True, dtype=np.float64,
                  handle_unknown='error'):
         self.categories = categories
@@ -282,6 +309,7 @@ def __init__(self, n_values=None, categorical_features=None,
         self.handle_unknown = handle_unknown
         self.n_values = n_values
         self.categorical_features = categorical_features
+        self.drop = drop
 
     # Deprecated attributes
 
@@ -354,20 +382,39 @@ def _handle_deprecations(self, X):
                     self._legacy_mode = False
                     self._categories = 'auto'
                 else:
-                    msg = (
-                        "The handling of integer data will change in version "
-                        "0.22. Currently, the categories are determined "
-                        "based on the range [0, max(values)], while in the "
-                        "future they will be determined based on the unique "
-                        "values.\nIf you want the future behaviour and "
-                        "silence this warning, you can specify "
-                        "\"categories='auto'\".\n"
-                        "In case you used a LabelEncoder before this "
-                        "OneHotEncoder to convert the categories to integers, "
-                        "then you can now use the OneHotEncoder directly."
-                    )
-                    warnings.warn(msg, FutureWarning)
-                    self._legacy_mode = True
+                    if self.drop is None:
+                        msg = (
+                            "The handling of integer data will change in "
+                            "version 0.22. Currently, the categories are "
+                            "determined based on the range "
+                            "[0, max(values)], while in the future they "
+                            "will be determined based on the unique "
+                            "values.\nIf you want the future behaviour "
+                            "and silence this warning, you can specify "
+                            "\"categories='auto'\".\n"
+                            "In case you used a LabelEncoder before this "
+                            "OneHotEncoder to convert the categories to "
+                            "integers, then you can now use the "
+                            "OneHotEncoder directly."
+                        )
+                        warnings.warn(msg, FutureWarning)
+                        self._legacy_mode = True
+                        self._n_values = 'auto'
+                    else:
+                        msg = (
+                            "The handling of integer data will change in "
+                            "version 0.22. Currently, the categories are "
+                            "determined based on the range "
+                            "[0, max(values)], while in the future they "
+                            "will be determined based on the unique "
+                            "values.\n The old behavior is not compatible "
+                            "with the `drop` paramenter. Instead, you "
+                            "must manually specify \"categories='auto'\" "
+                            "if you wish to use the `drop` parameter on "
+                            "an array of entirely integer data. This will "
+                            "enable the future behavior."
+                        )
+                        raise ValueError(msg)
 
         # if user specified categorical_features -> always use legacy mode
         if self.categorical_features is not None:
@@ -399,6 +446,21 @@ def _handle_deprecations(self, X):
         else:
             self._categorical_features = 'all'
 
+        # Prevents new drop functionality from being used in legacy mode
+        if self._legacy_mode and self.drop is not None:
+            raise ValueError(
+                "The `categorical_features` and `n_values` keywords "
+                "are deprecated, and cannot be used together "
+                "with 'drop'.")
+
+        # If we have both dropped columns and ignored unknown
+        # values, there will be ambiguous cells. This creates difficulties
+        # in interpreting the model.
+        if self.drop is not None and self.handle_unknown != 'error':
+            raise ValueError(
+                "`handle_unknown` cannot be 'ignore' when the drop parameter "
+                "is specified, as this will create ambiguous cells")
+
     def fit(self, X, y=None):
         """Fit OneHotEncoder to X.
 
@@ -425,6 +487,44 @@ def fit(self, X, y=None):
             return self
         else:
             self._fit(X, handle_unknown=self.handle_unknown)
+            if self.drop is None:
+                self.drop_idx_ = None
+            elif (isinstance(self.drop, six.string_types) and
+                    self.drop == 'first'):
+                self.drop_idx_ = np.zeros(len(self.categories_),
+                                          dtype=np.int8)
+            elif not isinstance(self.drop, six.string_types):
+                try:
+                    self.drop = np.asarray(self.drop, dtype=object)
+                    droplen = len(self.drop)
+                except (ValueError, TypeError):
+                    msg = ("Wrong input for parameter `drop`. Expected "
+                           "'first', None or array of objects, got {}")
+                    raise ValueError(msg.format(type(self.drop)))
+                if droplen != len(self.categories_):
+                    msg = ("`drop` should have length equal to the number "
+                           "of features ({}), got {}")
+                    raise ValueError(msg.format(len(self.categories_),
+                                                len(self.drop)))
+                missing_drops = [(i, val) for i, val in enumerate(self.drop)
+                                 if val not in self.categories_[i] and
+                                 val is not None]
+                if any(missing_drops):
+                    msg = ("The following categories were supposed to be "
+                           "dropped, but were not found in the training "
+                           "data.\n{}".format(
+                               "\n".join(
+                                    ["Category: {}, Feature: {}".format(c, v)
+                                        for c, v in missing_drops])))
+                    raise ValueError(msg)
+                self.drop_idx_ = np.array([np.where(cat_list == val)[0][0]
+                                           if val in cat_list else None
+                                           for (val, cat_list) in
+                                           zip(self.drop, self.categories_)])
+            else:
+                msg = ("Wrong input for parameter `drop`. Expected "
+                       "'first', None or array of objects, got {}")
+                raise ValueError(msg.format(type(self.drop)))
             return self
 
     def _legacy_fit_transform(self, X):
@@ -571,11 +671,23 @@ def _transform_new(self, X):
 
         X_int, X_mask = self._transform(X, handle_unknown=self.handle_unknown)
 
+        if self.drop is not None:
+            max_val = max(len(cats) for cats in self.categories_)
+            to_drop = np.array([pos if pos is not None else max_val
+                                for pos in self.drop_idx_]).reshape(1, -1)
+            keep_cells = X_int != to_drop
+            X_mask &= keep_cells
+            X_int[X_int > to_drop] -= 1
+            n_values = [len(cats) - 1 if drop_val is not None
+                        else len(cats)
+                        for cats, drop_val in
+                        zip(self.categories_, self.drop_idx_)]
+        else:
+            n_values = [cats.shape[0] for cats in self.categories_]
+
         mask = X_mask.ravel()
-        n_values = [cats.shape[0] for cats in self.categories_]
         n_values = np.array([0] + n_values)
         feature_indices = np.cumsum(n_values)
-
         indices = (X_int + feature_indices[:-1]).ravel()[mask]
         indptr = X_mask.sum(axis=1).cumsum()
         indptr = np.insert(indptr, 0, 0)
@@ -635,7 +747,13 @@ def inverse_transform(self, X):
 
         n_samples, _ = X.shape
         n_features = len(self.categories_)
-        n_transformed_features = sum([len(cats) for cats in self.categories_])
+        if self.drop is None:
+            n_transformed_features = sum(len(cats)
+                                         for cats in self.categories_)
+        else:
+            n_transformed_features = sum(len(cats) - (drop_idx is not None)
+                                         for cats, drop_idx in
+                                         zip(self.categories_, self.drop_idx_))
 
         # validate shape of passed X
         msg = ("Shape of the passed X data is not correct. Expected {0} "
@@ -651,17 +769,29 @@ def inverse_transform(self, X):
         found_unknown = {}
 
         for i in range(n_features):
-            n_categories = len(self.categories_[i])
+            if (self.drop is None) or self.drop_idx_[i] is None:
+                cats = self.categories_[i]
+            else:
+                cats = np.delete(self.categories_[i], self.drop_idx_[i])
+            n_categories = len(cats)
+            if n_categories == 0:
+                X_tr[:, i] = self.categories_[i][self.drop_idx_[i]]
+                j += n_categories
+                continue
             sub = X[:, j:j + n_categories]
-
             # for sparse X argmax returns 2D matrix, ensure 1D array
             labels = np.asarray(_argmax(sub, axis=1)).flatten()
-            X_tr[:, i] = self.categories_[i][labels]
-
-            if self.handle_unknown == 'ignore':
-                # ignored unknown categories: we have a row of all zero's
-                unknown = np.asarray(sub.sum(axis=1) == 0).flatten()
-                if unknown.any():
+            X_tr[:, i] = cats[labels]
+            unknown = np.asarray(sub.sum(axis=1) == 0).flatten()
+            if unknown.any():
+                '''
+                In this case, we can safely assume that all of the nulls in
+                each column are the dropped value
+                '''
+                if self.drop is not None:
+                    X_tr[unknown, i] = self.categories_[i][self.drop_idx_[i]]
+                elif self.handle_unknown == 'ignore':
+                    # ignored unknown categories: we have a row of all zero's
                     found_unknown[i] = unknown
 
             j += n_categories
diff --git a/sklearn/preprocessing/tests/test_encoders.py b/sklearn/preprocessing/tests/test_encoders.py
index 93b1de018d2e9..14e28b09f2e04 100644
--- a/sklearn/preprocessing/tests/test_encoders.py
+++ b/sklearn/preprocessing/tests/test_encoders.py
@@ -96,6 +96,20 @@ def test_one_hot_encoder_sparse():
         enc.fit([[0], [1]])
     assert_raises(ValueError, enc.transform, [[0], [-1]])
 
+    with ignore_warnings(category=(DeprecationWarning, FutureWarning)):
+        enc = OneHotEncoder(drop='first', n_values=1)
+        for method in (enc.fit, enc.fit_transform):
+            assert_raises_regex(
+                ValueError,
+                'The `categorical_features` and `n_values` keywords ',
+                method, [[0], [-1]])
+
+            enc = OneHotEncoder(drop='first', categorical_features='all')
+            assert_raises_regex(
+                ValueError,
+                'The `categorical_features` and `n_values` keywords ',
+                method, [[0], [-1]])
+
 
 def test_one_hot_encoder_dense():
     # check for sparse=False
@@ -278,7 +292,7 @@ def test_one_hot_encoder_no_categorical_features():
     enc = OneHotEncoder(categorical_features=cat)
     with ignore_warnings(category=(DeprecationWarning, FutureWarning)):
         X_tr = enc.fit_transform(X)
-    expected_features = np.array(list(), dtype='object')
+    expected_features = np.array([], dtype='object')
     assert_array_equal(X, X_tr)
     assert_array_equal(enc.get_feature_names(), expected_features)
     assert enc.categories_ == []
@@ -373,21 +387,25 @@ def test_one_hot_encoder(X):
     assert_allclose(Xtr.toarray(), [[0, 1, 1, 0,  1], [1, 0, 0, 1, 1]])
 
 
-def test_one_hot_encoder_inverse():
-    for sparse_ in [True, False]:
-        X = [['abc', 2, 55], ['def', 1, 55], ['abc', 3, 55]]
-        enc = OneHotEncoder(sparse=sparse_)
-        X_tr = enc.fit_transform(X)
-        exp = np.array(X, dtype=object)
-        assert_array_equal(enc.inverse_transform(X_tr), exp)
+@pytest.mark.parametrize('sparse_', [False, True])
+@pytest.mark.parametrize('drop', [None, 'first'])
+def test_one_hot_encoder_inverse(sparse_, drop):
+    X = [['abc', 2, 55], ['def', 1, 55], ['abc', 3, 55]]
+    enc = OneHotEncoder(sparse=sparse_, drop=drop)
+    X_tr = enc.fit_transform(X)
+    exp = np.array(X, dtype=object)
+    assert_array_equal(enc.inverse_transform(X_tr), exp)
 
-        X = [[2, 55], [1, 55], [3, 55]]
-        enc = OneHotEncoder(sparse=sparse_, categories='auto')
-        X_tr = enc.fit_transform(X)
-        exp = np.array(X)
-        assert_array_equal(enc.inverse_transform(X_tr), exp)
+    X = [[2, 55], [1, 55], [3, 55]]
+    enc = OneHotEncoder(sparse=sparse_, categories='auto',
+                        drop=drop)
+    X_tr = enc.fit_transform(X)
+    exp = np.array(X)
+    assert_array_equal(enc.inverse_transform(X_tr), exp)
 
+    if drop is None:
         # with unknown categories
+        # drop_first is incompatible with handle_unknown=ignore
         X = [['abc', 2, 55], ['def', 1, 55], ['abc', 3, 55]]
         enc = OneHotEncoder(sparse=sparse_, handle_unknown='ignore',
                             categories=[['abc', 'def'], [1, 2],
@@ -407,10 +425,10 @@ def test_one_hot_encoder_inverse():
         exp[:, 1] = None
         assert_array_equal(enc.inverse_transform(X_tr), exp)
 
-        # incorrect shape raises
-        X_tr = np.array([[0, 1, 1], [1, 0, 1]])
-        msg = re.escape('Shape of the passed X data is not correct')
-        assert_raises_regex(ValueError, msg, enc.inverse_transform, X_tr)
+    # incorrect shape raises
+    X_tr = np.array([[0, 1, 1], [1, 0, 1]])
+    msg = re.escape('Shape of the passed X data is not correct')
+    assert_raises_regex(ValueError, msg, enc.inverse_transform, X_tr)
 
 
 @pytest.mark.parametrize("X, cat_exp, cat_dtype", [
@@ -687,3 +705,63 @@ def test_one_hot_encoder_warning():
     enc = OneHotEncoder()
     X = [['Male', 1], ['Female', 3]]
     np.testing.assert_no_warnings(enc.fit_transform, X)
+
+    enc = OneHotEncoder(drop=['def', 3, 56])
+    X = [['abc', 2, 55], ['def', 1, 55], ['def', 3, 56]]
+    exp = np.array([[1., 0., 1., 1.],
+                    [0., 1., 0., 1.],
+                    [0., 0., 0., 0.]], dtype='float64')
+    assert_array_equal(enc.fit_transform(X).toarray(), exp)
+
+
+def test_one_hot_encoder_none_drop():
+    enc = OneHotEncoder(drop=['def', 3, None])
+    X = [['abc', 2, 55], ['def', 1, 55], ['def', 3, 56]]
+    exp = np.array([[1., 0., 1., 1., 0.],
+                    [0., 1., 0., 1., 0.],
+                    [0., 0., 0., 0., 1.]], dtype='float64')
+    assert_array_equal(enc.fit_transform(X).toarray(), exp)
+
+
+def test_one_hot_encoder_invalid_params():
+    enc = OneHotEncoder(drop='second')
+    assert_raises_regex(
+        ValueError,
+        "Wrong input for parameter `drop`.",
+        enc.fit, [["Male"], ["Female"]])
+
+    enc = OneHotEncoder(handle_unknown='ignore', drop='first')
+    assert_raises_regex(
+        ValueError,
+        "`handle_unknown` cannot be 'ignore'",
+        enc.fit, [["Male"], ["Female"]])
+
+    enc = OneHotEncoder(drop='first')
+    assert_raises_regex(
+        ValueError,
+        "The handling of integer data will change in version",
+        enc.fit, [[1], [2]])
+
+    enc = OneHotEncoder(drop='first', categories='auto')
+    assert_no_warnings(enc.fit_transform, [[1], [2]])
+
+    enc = OneHotEncoder(drop=np.asarray('b', dtype=object))
+    assert_raises_regex(
+        ValueError,
+        "Wrong input for parameter `drop`.",
+        enc.fit, [['abc', 2, 55], ['def', 1, 55], ['def', 3, 59]])
+
+    enc = OneHotEncoder(drop=['ghi', 3, 59])
+    assert_raises_regex(
+        ValueError,
+        "The following categories were supposed",
+        enc.fit, [['abc', 2, 55], ['def', 1, 55], ['def', 3, 59]])
+
+
+@pytest.mark.parametrize('drop', [['abc', 3], ['abc', 3, 41, 'a']])
+def test_invalid_drop_length(drop):
+    enc = OneHotEncoder(drop=drop)
+    assert_raises_regex(
+        ValueError,
+        "`drop` should have length equal to the number",
+        enc.fit, [['abc', 2, 55], ['def', 1, 55], ['def', 3, 59]])


Code review comments:
diff_hunk: @@ -571,11 +671,23 @@ def _transform_new(self, X):
 
         X_int, X_mask = self._transform(X, handle_unknown=self.handle_unknown)
 
+        if self.drop is not None:
+            max_val = max(len(cats) for cats in self.categories_)
+            to_drop = np.array([pos if pos is not None else max_val
path: sklearn/preprocessing/_encoders.py
text: @user1:
I prefer not negating if-else operator condition. `max_val if pos is None else pos`

diff_hunk: @@ -651,17 +769,29 @@ def inverse_transform(self, X):
         found_unknown = {}
 
         for i in range(n_features):
-            n_categories = len(self.categories_[i])
+            if (self.drop is None) or self.drop_idx_[i] is None:
+                cats = self.categories_[i]
+            else:
+                cats = np.delete(self.categories_[i], self.drop_idx_[i])
+            n_categories = len(cats)
+            if n_categories == 0:
+                X_tr[:, i] = self.categories_[i][self.drop_idx_[i]]
+                j += n_categories
+                continue
             sub = X[:, j:j + n_categories]
-
             # for sparse X argmax returns 2D matrix, ensure 1D array
             labels = np.asarray(_argmax(sub, axis=1)).flatten()
-            X_tr[:, i] = self.categories_[i][labels]
-
-            if self.handle_unknown == 'ignore':
-                # ignored unknown categories: we have a row of all zero's
-                unknown = np.asarray(sub.sum(axis=1) == 0).flatten()
-                if unknown.any():
+            X_tr[:, i] = cats[labels]
+            unknown = np.asarray(sub.sum(axis=1) == 0).flatten()
path: sklearn/preprocessing/_encoders.py
text: @user1:
I think you want `~sub.any(axis=1)`

@author:
Hrm, this doesn't play nicely with CSR matrices. Do you think it's worth converting a sparse representation to a dense one to use this?

diff_hunk: @@ -425,6 +487,44 @@ def fit(self, X, y=None):
             return self
         else:
             self._fit(X, handle_unknown=self.handle_unknown)
+            if self.drop is None:
path: sklearn/preprocessing/_encoders.py
text: @user1:
let's pull this out into a method. `self.drop_idx_ = self._compute_drop_idx()`

diff_hunk: @@ -207,6 +224,10 @@ class OneHotEncoder(_BaseEncoder):
         (in order of the features in X and corresponding with the output
         of ``transform``).
path: sklearn/preprocessing/_encoders.py
text: @user1:
Note that `categories_` includes the dropped category.

diff_hunk: @@ -651,17 +769,29 @@ def inverse_transform(self, X):
         found_unknown = {}
 
         for i in range(n_features):
-            n_categories = len(self.categories_[i])
+            if (self.drop is None) or self.drop_idx_[i] is None:
path: sklearn/preprocessing/_encoders.py
text: @user1:
rm unnecessary parentheses

diff_hunk: @@ -373,21 +387,25 @@ def test_one_hot_encoder(X):
     assert_allclose(Xtr.toarray(), [[0, 1, 1, 0,  1], [1, 0, 0, 1, 1]])
 
 
-def test_one_hot_encoder_inverse():
-    for sparse_ in [True, False]:
-        X = [['abc', 2, 55], ['def', 1, 55], ['abc', 3, 55]]
-        enc = OneHotEncoder(sparse=sparse_)
-        X_tr = enc.fit_transform(X)
-        exp = np.array(X, dtype=object)
-        assert_array_equal(enc.inverse_transform(X_tr), exp)
+@pytest.mark.parametrize('sparse_', [False, True])
+@pytest.mark.parametrize('drop', [None, 'first'])
+def test_one_hot_encoder_inverse(sparse_, drop):
+    X = [['abc', 2, 55], ['def', 1, 55], ['abc', 3, 55]]
+    enc = OneHotEncoder(sparse=sparse_, drop=drop)
+    X_tr = enc.fit_transform(X)
+    exp = np.array(X, dtype=object)
+    assert_array_equal(enc.inverse_transform(X_tr), exp)
 
-        X = [[2, 55], [1, 55], [3, 55]]
-        enc = OneHotEncoder(sparse=sparse_, categories='auto')
-        X_tr = enc.fit_transform(X)
-        exp = np.array(X)
-        assert_array_equal(enc.inverse_transform(X_tr), exp)
+    X = [[2, 55], [1, 55], [3, 55]]
+    enc = OneHotEncoder(sparse=sparse_, categories='auto',
+                        drop=drop)
+    X_tr = enc.fit_transform(X)
+    exp = np.array(X)
+    assert_array_equal(enc.inverse_transform(X_tr), exp)
 
+    if drop is None:
         # with unknown categories
+        # drop_first is incompatible with handle_unknown=ignore
path: sklearn/preprocessing/tests/test_encoders.py
text: @user1:
`drop_first` -> `drop`

diff_hunk: @@ -687,3 +705,63 @@ def test_one_hot_encoder_warning():
     enc = OneHotEncoder()
     X = [['Male', 1], ['Female', 3]]
     np.testing.assert_no_warnings(enc.fit_transform, X)
+
+    enc = OneHotEncoder(drop=['def', 3, 56])
path: sklearn/preprocessing/tests/test_encoders.py
text: @user1:
what's this got to do with testing the warning?

@author:
I think I accidentally deleted the header line for the next test!