Repository: scikit-learn/scikit-learn
Pull request: [MRG + 1] ENH: new CategoricalEncoder class
This is currently simply a rebase of PR https://github.com/scikit-learn/scikit-learn/pull/6559 by @vighneshbirodkar . 

Some context: this PR https://github.com/scikit-learn/scikit-learn/pull/6559 was the first of a series of PRs related to this. This PR added a CategoricalEncoder. Then it was decided, instead of adding a new class, to add this functionality to the existing OneHotEncoder: https://github.com/scikit-learn/scikit-learn/pull/8793 and https://github.com/scikit-learn/scikit-learn/pull/7327, and recently taken up by @stephen-hoover in https://github.com/scikit-learn/scikit-learn/pull/8793.

At the sprint we discussed this, and @amueller put a summary of that in https://github.com/scikit-learn/scikit-learn/pull/8793#issuecomment-307553326. 
The main reason not to add this in OneHotEncoder is that this is fundamentally different behaviour (OneHotEncoder determines the categories based on the range of the positive integer values passed, the new CategoricalEncoder would determine it based on unique values), and that almost all keyword, attributes and behaviour of the current OneHotEncoder would be deprecated, which makes the implementation to do this in one class (deprecated + new behaviour) overly complex.
    
The basics already work nicely with the rebased PR:

```py
In [30]: cat = CategoricalEncoder()

In [31]: X = np.array([['a', 'b', 'a', 'c'], [0, 1, 0, 1]], dtype=object).T

In [32]: cat.fit_transform(X).toarray()
Out[32]: 
array([[ 1.,  0.,  0.,  1.,  0.],
       [ 0.,  1.,  0.,  0.,  1.],
       [ 1.,  0.,  0.,  1.,  0.],
       [ 0.,  0.,  1.,  0.,  1.]])
```

Some changes I would like to make to the current PR:

- add some more tests
- rename 'classes' keyword to 'categories' (IMO this is more in line with the name of the class 'CategoricalEncoder')
- possibly remove the `categorical_features` keyword (the ability to select certain columns) for now to keep it more simple, as this can always be achieved in combination with the ColumnTransformer
- I would like to add a `categories_` attribute that is a dict of {column number/name : [categories]}. And maybe the underlying LabelEncoders can be hidden from the users (currently stored in the `label_encoders_` attribute)
- add support for pandas DataFrames (they already work, but would be nice to keep column -> categories information, see previous)
- don't deprecate OneHotEncoder for now (we can leave this for a separate discussion) 
- move to sklearn.experimental (if we keep to this for the ColumnTransformer)
- add `get_feature_names()` method ?


But before doing that, I wanted to check if we agree on this way forward (separate CategoricalEncoder class). @jnothman are you OK with this or still in favor of doing the changes in the OneHotEncoder?

If we want to keep the name OneHotEncoder, another option would be to implement the 'new' OneHotEncoder in eg a 'sklearn.future' module, so people can still move to it gradually and the current one can be deprecated, but keeping the implementations separate.

Closes #7375, closes #7327, closes #4920, closes #3956

Related issue that can possibly be closed as well: #3599, https://github.com/scikit-learn/scikit-learn/issues/8136

Problem statement:
Pipeline doesn't work with Label Encoder
I've found that I cannot use pipelines if I wish to use the label encoder. In the following I wish to build a pipeline that first encodes the label and then constructs a one-hot encoding from that labelling. 

```
from sklearn.preprocessing import  OneHotEncoder, LabelEncoder
from sklearn.pipeline import make_pipeline
import numpy as np

X = np.array(['cat', 'dog', 'cow', 'cat', 'cow', 'dog'])

enc = LabelEncoder()
hot = OneHotEncoder()

pipe = make_pipeline(enc, hot)
pipe.fit_transform(X)
```

However, the following error is returned:

```
lib/python2.7/site-packages/sklearn/pipeline.pyc in _pre_transform(self, X, y, **fit_params)
    117         for name, transform in self.steps[:-1]:
    118             if hasattr(transform, "fit_transform"):
--> 119                 Xt = transform.fit_transform(Xt, y, **fit_params_steps[name])
    120             else:
    121                 Xt = transform.fit(Xt, y, **fit_params_steps[name]) \

TypeError: fit_transform() takes exactly 2 arguments (3 given)
```

It seems that the problem is that the fit method for label encoder only takes a y argument, whereas the pipeline assumes that it will take an X and an optional y. 


OneHotEncoder should accept string values
How hard would it be to hash incoming string values, or use the LabelEncoder internally?

I'd be willing to contribute some code for this if there are no outstanding design blockers.

/cc @jnothman

Hints from issue discussion:
Do you mean to use `LabelBinarizer` instead? This has been a subject of complaint previously, and we should probably document this more clearly. The label "transformers" should probably not have the same interface as data transformers.

It is a bit odd that we don't have these for data, though. We have the `DictVectorizer` and `OneHotEncoder` but I don't see a way how to perform the operation here without creating a dict first.
In particular, the docs: http://scikit-learn.org/dev/modules/preprocessing.html#encoding-categorical-features leave out exactly the step that is breaking here.... that is not super great....

Is there a good reason why one hot encoder can't just work with string values**? In my experience, the task of one hot encoding a string valued feature is incredibly common, and it seems tedious to have to use to preprocessing transformers to accomplish it. I'd be happy to contribute code for it.

*\* By work with string values I mean:

```
string_feature = np.array(['cat', 'dog', 'cat', 'ferret'])
```

would be transformed to:

```
[[1, 0, 0],
 [0, 1, 0],
 [1, 0, 0]
 [0, 0, 1]]
```

@mkneierV, That particular case happens to work fine with `CountVectorizer`. See also #4920, but as you note the description of "work with strings" is underspecified: in general, I think we must require 2d input for `OneHotEncoder`. So it could encode `[['cat'], ['dog'], ['cat'], ['ferret']]` as above `[['cat', 'cat'], ['dog', 'dog'], ['cat', 'cat'], ['ferret', 'ferret']]` as the same duplicated horizontally.

+1. Currently this can only be done by using (the not merged) FunctionTransformer and a FeatureUnion :-/

This should really work for arbitrary categorical matrices (and mixed type data).

This is bitting me too. I constantly have to one-hot-encode string features. Since these features usually come from a Pandas dataframe, I don't see why should we force users to transform the columns to a dictionary to be able to use the `DictVectorizer`.

I guess one should never mix label transformers (which usually expect a single 1-dimensional argument) and data transformers, but I think people (including me) do this because there are not equivalent data transformers for certain label ones, like `LabelEncoder` or `LabelBinarizer`.

I agree. The solution is to improve `OneHotEncoder` I think. See #4920. There was also a related issue that I can't find any more (we should be able to encode multiple string-valued columns. Maybe @jnothman remembers?

I fount a way to go around the problem by using the `CountVectorizer` which can turn those strings into a binary representation directly:

```
from sklearn.feature_extraction.text import CountVectorizer
CountVectorizer().fit_transform(X).todense()
Out[]: 
matrix([[1, 0, 0],
        [0, 0, 1],
        [0, 1, 0],
        [1, 0, 0],
        [0, 1, 0],
        [0, 0, 1]], dtype=int64)
```

I think this is a good idea (and I think a duplicate issue).

Related to #4883.

Looks like it.

@vighneshbirodkar is working on this (I think)

Yes. But I am waiting for the merge of #5270

@vighneshbirodkar , I have a very hacky equivalent of this running that relies on pandas.get_dummies in a custom pipeline element with a very janky fit and transform (so it requires a dataframe as an input, I've read previously in @amueller 's comments that this is ill advised, but did not heed his warning, sorry); I'm sure you probably have a much more usable and intelligent solution, but if I can help out with any of what I've written/thought about on it, please let me know.

Commit message: Added CategoricalEncoder class - deprecating OneHotEncoder

Patch to review:
diff --git a/doc/modules/classes.rst b/doc/modules/classes.rst
index 950a9320dd1af..601d41b682791 100644
--- a/doc/modules/classes.rst
+++ b/doc/modules/classes.rst
@@ -1197,7 +1197,7 @@ See the :ref:`metrics` section of the user guide for further details.
    preprocessing.MaxAbsScaler
    preprocessing.MinMaxScaler
    preprocessing.Normalizer
-   preprocessing.OneHotEncoder
+   preprocessing.CategoricalEncoder
    preprocessing.PolynomialFeatures
    preprocessing.QuantileTransformer
    preprocessing.RobustScaler
diff --git a/doc/modules/preprocessing.rst b/doc/modules/preprocessing.rst
index a4e1364a85ae6..dd85a6c6faa98 100644
--- a/doc/modules/preprocessing.rst
+++ b/doc/modules/preprocessing.rst
@@ -461,38 +461,45 @@ not desired (i.e. the set of browsers was ordered arbitrarily).
 
 One possibility to convert categorical features to features that can be used
 with scikit-learn estimators is to use a one-of-K or one-hot encoding, which is
-implemented in :class:`OneHotEncoder`.  This estimator transforms each
+implemented in :class:`CategoricalEncoder`.  This estimator transforms each
 categorical feature with ``m`` possible values into ``m`` binary features, with
 only one active.
 
 Continuing the example above::
 
-  >>> enc = preprocessing.OneHotEncoder()
-  >>> enc.fit([[0, 0, 3], [1, 1, 0], [0, 2, 1], [1, 0, 2]])  # doctest: +ELLIPSIS
-  OneHotEncoder(categorical_features='all', dtype=<... 'numpy.float64'>,
-         handle_unknown='error', n_values='auto', sparse=True)
-  >>> enc.transform([[0, 1, 3]]).toarray()
-  array([[ 1.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  1.]])
+  >>> enc = preprocessing.CategoricalEncoder()
+  >>> X = [['male', 'from US', 'uses Safari'], ['female', 'from Europe', 'uses Firefox']]
+  >>> enc.fit(X)  # doctest: +ELLIPSIS
+  CategoricalEncoder(categorical_features='all', classes='auto',
+            dtype=<... 'numpy.float64'>, handle_unknown='error',
+            sparse=True)
+  >>> enc.transform([['female', 'from US', 'uses Safari']]).toarray()
+  array([[ 1.,  0.,  0.,  1.,  0.,  1.]])
+
 
 By default, how many values each feature can take is inferred automatically from the dataset.
-It is possible to specify this explicitly using the parameter ``n_values``.
+It is possible to specify this explicitly using the parameter ``classes``.
 There are two genders, three possible continents and four web browsers in our
 dataset.
-Then we fit the estimator, and transform a data point.
-In the result, the first two numbers encode the gender, the next set of three
-numbers the continent and the last four the web browser.
 
 Note that, if there is a possibilty that the training data might have missing categorical
-features, one has to explicitly set ``n_values``. For example,
-
-    >>> enc = preprocessing.OneHotEncoder(n_values=[2, 3, 4])
-    >>> # Note that there are missing categorical values for the 2nd and 3rd
-    >>> # features
-    >>> enc.fit([[1, 2, 3], [0, 2, 0]])  # doctest: +ELLIPSIS
-    OneHotEncoder(categorical_features='all', dtype=<... 'numpy.float64'>,
-           handle_unknown='error', n_values=[2, 3, 4], sparse=True)
-    >>> enc.transform([[1, 0, 0]]).toarray()
-    array([[ 0.,  1.,  1.,  0.,  0.,  1.,  0.,  0.,  0.]])
+features, one has to explicitly set ``classes``. For example,
+
+    >>> genders = ['male', 'female']
+    >>> locations = ['from Europe', 'from US', 'from Africa', 'from Asia']
+    >>> browsers = ['uses Safari', 'uses Firefox', 'uses IE', 'uses Chrome']
+    >>> enc = preprocessing.CategoricalEncoder(classes=[genders, locations, browsers])
+    >>> # Note that for there are missing categorical values for the 2nd and 3rd
+    >>> # feature
+    >>> X = [['male', 'from US', 'uses Safari'], ['female', 'from Europe', 'uses Firefox']]
+    >>> enc.fit(X) # doctest: +ELLIPSIS
+    CategoricalEncoder(categorical_features='all',
+              classes=[...],
+              dtype=<... 'numpy.float64'>, handle_unknown='error',
+              sparse=True)
+
+    >>> enc.transform([['female', 'from Asia', 'uses Chrome']]).toarray()
+    array([[ 1.,  0.,  0.,  0.,  0.,  1.,  1.,  0.,  0.,  0.]])
 
 See :ref:`dict_feature_extraction` for categorical features that are represented
 as a dict, not as integers.
diff --git a/examples/ensemble/plot_feature_transformation.py b/examples/ensemble/plot_feature_transformation.py
index e004c167e67af..50347de57f12b 100644
--- a/examples/ensemble/plot_feature_transformation.py
+++ b/examples/ensemble/plot_feature_transformation.py
@@ -34,7 +34,7 @@
 from sklearn.linear_model import LogisticRegression
 from sklearn.ensemble import (RandomTreesEmbedding, RandomForestClassifier,
                               GradientBoostingClassifier)
-from sklearn.preprocessing import OneHotEncoder
+from sklearn.preprocessing import CategoricalEncoder
 from sklearn.model_selection import train_test_split
 from sklearn.metrics import roc_curve
 from sklearn.pipeline import make_pipeline
@@ -62,7 +62,7 @@
 
 # Supervised transformation based on random forests
 rf = RandomForestClassifier(max_depth=3, n_estimators=n_estimator)
-rf_enc = OneHotEncoder()
+rf_enc = CategoricalEncoder()
 rf_lm = LogisticRegression()
 rf.fit(X_train, y_train)
 rf_enc.fit(rf.apply(X_train))
@@ -72,7 +72,7 @@
 fpr_rf_lm, tpr_rf_lm, _ = roc_curve(y_test, y_pred_rf_lm)
 
 grd = GradientBoostingClassifier(n_estimators=n_estimator)
-grd_enc = OneHotEncoder()
+grd_enc = CategoricalEncoder()
 grd_lm = LogisticRegression()
 grd.fit(X_train, y_train)
 grd_enc.fit(grd.apply(X_train)[:, :, 0])
diff --git a/sklearn/feature_extraction/dict_vectorizer.py b/sklearn/feature_extraction/dict_vectorizer.py
index 53804ed83ac45..d01067efba19d 100644
--- a/sklearn/feature_extraction/dict_vectorizer.py
+++ b/sklearn/feature_extraction/dict_vectorizer.py
@@ -39,7 +39,8 @@ class DictVectorizer(BaseEstimator, TransformerMixin):
     However, note that this transformer will only do a binary one-hot encoding
     when feature values are of type string. If categorical features are
     represented as numeric values such as int, the DictVectorizer can be
-    followed by OneHotEncoder to complete binary one-hot encoding.
+    followed by :class:`sklearn.preprocessing.CategoricalEncoder` to complete
+    binary one-hot encoding.
 
     Features that do not occur in a sample (mapping) will have a zero value
     in the resulting array/matrix.
@@ -88,8 +89,8 @@ class DictVectorizer(BaseEstimator, TransformerMixin):
     See also
     --------
     FeatureHasher : performs vectorization using only a hash function.
-    sklearn.preprocessing.OneHotEncoder : handles nominal/categorical features
-      encoded as columns of integers.
+    sklearn.preprocessing.CategoricalEncoder : handles nominal/categorical
+      features encoded as columns of arbitraty data types.
     """
 
     def __init__(self, dtype=np.float64, separator="=", sparse=True,
diff --git a/sklearn/preprocessing/__init__.py b/sklearn/preprocessing/__init__.py
index 2b105709ffe08..0f5054e57f608 100644
--- a/sklearn/preprocessing/__init__.py
+++ b/sklearn/preprocessing/__init__.py
@@ -22,6 +22,7 @@
 from .data import minmax_scale
 from .data import quantile_transform
 from .data import OneHotEncoder
+from .data import CategoricalEncoder
 
 from .data import PolynomialFeatures
 
@@ -46,6 +47,7 @@
     'QuantileTransformer',
     'Normalizer',
     'OneHotEncoder',
+    'CategoricalEncoder',
     'RobustScaler',
     'StandardScaler',
     'add_dummy_feature',
diff --git a/sklearn/preprocessing/data.py b/sklearn/preprocessing/data.py
index c9de8a99a0f3d..3f84bfbae3ac8 100644
--- a/sklearn/preprocessing/data.py
+++ b/sklearn/preprocessing/data.py
@@ -29,6 +29,11 @@
                                  min_max_axis)
 from ..utils.validation import (check_is_fitted, check_random_state,
                                 FLOAT_DTYPES)
+from .label import LabelEncoder
+from ..utils.fixes import np_version
+from ..utils.deprecation import deprecated
+
+
 BOUNDS_THRESHOLD = 1e-7
 
 
@@ -1677,28 +1682,27 @@ def add_dummy_feature(X, value=1.0):
         return np.hstack((np.ones((n_samples, 1)) * value, X))
 
 
-def _transform_selected(X, transform, selected="all", copy=True):
-    """Apply a transform function to portion of selected features
-
+def _apply_selected(X, transform, selected="all", dtype=np.float, copy=True,
+                    return_val=True):
+    """Apply a function to portion of selected features
     Parameters
     ----------
-    X : {array-like, sparse matrix}, shape [n_samples, n_features]
+    X : {array, sparse matrix}, shape [n_samples, n_features]
         Dense array or sparse matrix.
-
     transform : callable
         A callable transform(X) -> X_transformed
-
     copy : boolean, optional
         Copy X even if it could be avoided.
-
     selected: "all" or array of indices or mask
         Specify which features to apply the transform to.
-
+    return_val : boolean, optional
+        Whether to return the transformed matrix. If not set `None` is
+        returned.
     Returns
     -------
-    X : array or sparse matrix, shape=(n_samples, n_features_new)
+        X : array or sparse matrix, shape=(n_samples, n_features_new)
     """
-    X = check_array(X, accept_sparse='csc', copy=copy, dtype=FLOAT_DTYPES)
+    X = check_array(X, accept_sparse='csc', copy=copy, dtype=None)
 
     if isinstance(selected, six.string_types) and selected == "all":
         return transform(X)
@@ -1721,14 +1725,16 @@ def _transform_selected(X, transform, selected="all", copy=True):
         return transform(X)
     else:
         X_sel = transform(X[:, ind[sel]])
-        X_not_sel = X[:, ind[not_sel]]
+        X_not_sel = X[:, ind[not_sel]].astype(dtype)
 
-        if sparse.issparse(X_sel) or sparse.issparse(X_not_sel):
-            return sparse.hstack((X_sel, X_not_sel))
-        else:
-            return np.hstack((X_sel, X_not_sel))
+        if return_val:
+            if sparse.issparse(X_sel) or sparse.issparse(X_not_sel):
+                return sparse.hstack((X_sel, X_not_sel))
+            else:
+                return np.hstack((X_sel, X_not_sel))
 
 
+@deprecated('`OneHotEncoder` is deprecated, use `CategoricalEncoder` instead.')
 class OneHotEncoder(BaseEstimator, TransformerMixin):
     """Encode categorical integer features using a one-hot aka one-of-K scheme.
 
@@ -1902,8 +1908,8 @@ def fit_transform(self, X, y=None):
         Equivalent to self.fit(X).transform(X), but more convenient and more
         efficient. See fit for the parameters, transform for the return value.
         """
-        return _transform_selected(X, self._fit_transform,
-                                   self.categorical_features, copy=True)
+        return _apply_selected(X, self._fit_transform, dtype=self.dtype,
+                               selected=self.categorical_features, copy=True)
 
     def _transform(self, X):
         """Assumes X contains only categorical features."""
@@ -1958,8 +1964,8 @@ def transform(self, X):
         X_out : sparse matrix if sparse=True else a 2-d array, dtype=int
             Transformed input.
         """
-        return _transform_selected(X, self._transform,
-                                   self.categorical_features, copy=True)
+        return _apply_selected(X, self._transform, dtype=self.dtype,
+                               selected=self.categorical_features, copy=True)
 
 
 class QuantileTransformer(BaseEstimator, TransformerMixin):
@@ -2440,3 +2446,189 @@ def quantile_transform(X, axis=0, n_quantiles=1000,
     else:
         raise ValueError("axis should be either equal to 0 or 1. Got"
                          " axis={}".format(axis))
+
+
+class CategoricalEncoder(BaseEstimator, TransformerMixin):
+    """Encode categorical features using a one-hot aka one-of-K scheme.
+
+    The input to this transformer should be a matrix of integers or strings,
+    denoting the values taken on by categorical (discrete) features. The
+    output will be a sparse matrix where each column corresponds to one
+    possible value of one feature.
+
+    This encoding is needed for feeding categorical data to many scikit-learn
+    estimators, notably linear models and SVMs with the standard kernels.
+    Read more in the :ref:`User Guide <preprocessing_categorical_features>`.
+
+    Parameters
+    ----------
+    classes : 'auto', 2D array of ints or strings or both.
+        Values per feature.
+
+        - 'auto' : Determine classes automatically from the training data.
+        - array: ``classes[i]`` holds the classes expected in the ith column.
+
+    categorical_features : 'all' or array of indices or mask
+        Specify what features are treated as categorical.
+
+        - 'all' (default): All features are treated as categorical.
+        - array of indices: Array of categorical feature indices.
+        - mask: Array of length n_features and with dtype=bool.
+        Non-categorical features are always stacked to the right of the matrix.
+
+    dtype : number type, default=np.float
+        Desired dtype of output.
+
+    sparse : boolean, default=True
+        Will return sparse matrix if set True else will return an array.
+
+    handle_unknown : str, 'error' or 'ignore'
+        Whether to raise an error or ignore if a unknown categorical feature is
+        present during transform.
+
+    Attributes
+    ----------
+    label_encoders_ : list of size n_features.
+        The :class:`sklearn.preprocessing.LabelEncoder` objects used to encode
+        the features. ``self.label_encoders[i]_`` is the LabelEncoder object
+        used to encode the ith column. The unique features found on column
+        ``i`` can be accessed using ``self.label_encoders_[i].classes_``.
+
+    Examples
+    --------
+    Given a dataset with three features and two samples, we let the encoder
+    find the maximum value per feature and transform the data to a binary
+    one-hot encoding.
+    >>> from sklearn.preprocessing import CategoricalEncoder
+    >>> enc = CategoricalEncoder()
+    >>> enc.fit([[0, 0, 3], [1, 1, 0], [0, 2, 1], \
+[1, 0, 2]])  # doctest: +ELLIPSIS
+    CategoricalEncoder(categorical_features='all', classes='auto',
+              dtype=<... 'numpy.float64'>, handle_unknown='error',
+              sparse=True)
+    >>> enc.transform([[0, 1, 1]]).toarray()
+    array([[ 1.,  0.,  0.,  1.,  0.,  0.,  1.,  0.,  0.]])
+
+    See also
+    --------
+    sklearn.feature_extraction.DictVectorizer : performs a one-hot encoding of
+      dictionary items (also handles string-valued features).
+    sklearn.feature_extraction.FeatureHasher : performs an approximate one-hot
+      encoding of dictionary items or strings.
+    """
+
+    def __init__(self, classes='auto', categorical_features="all",
+                 dtype=np.float64, sparse=True, handle_unknown='error'):
+        self.classes = classes
+        self.categorical_features = categorical_features
+        self.dtype = dtype
+        self.sparse = sparse
+        self.handle_unknown = handle_unknown
+
+    def fit(self, X, y=None):
+        """Fit the CategoricalEncoder to X.
+
+        Parameters
+        ----------
+        X : array-like, shape [n_samples, n_feature]
+            Input array of type int.
+
+        Returns
+        -------
+        self
+        """
+
+        if self.handle_unknown not in ['error', 'ignore']:
+            template = ("handle_unknown should be either 'error' or "
+                        "'ignore', got %s")
+            raise ValueError(template % self.handle_unknown)
+
+        X = check_array(X, dtype=np.object, accept_sparse='csc')
+        n_samples, n_features = X.shape
+
+        _apply_selected(X, self._fit, dtype=self.dtype,
+                        selected=self.categorical_features, copy=True,
+                        return_val=False)
+        return self
+
+    def _fit(self, X):
+        "Assumes `X` contains only cetergorical features."
+
+        X = check_array(X, dtype=np.object)
+        n_samples, n_features = X.shape
+
+        self.label_encoders_ = [LabelEncoder() for i in range(n_features)]
+
+        for i in range(n_features):
+            le = self.label_encoders_[i]
+            if self.classes == 'auto':
+                le.fit(X[:, i])
+            else:
+                le.classes_ = np.array(self.classes[i])
+
+    def transform(self, X, y=None):
+        """Encode the selected categorical features using the one-hot scheme.
+        """
+        X = check_array(X, dtype=np.object)
+        return _apply_selected(X, self._transform, copy=True,
+                               selected=self.categorical_features)
+
+    def _transform(self, X):
+        "Assumes `X` contains only categorical features."
+
+        X = check_array(X, accept_sparse='csc', dtype=np.object)
+        n_samples, n_features = X.shape
+        X_int = np.zeros_like(X, dtype=np.int)
+        X_mask = np.ones_like(X, dtype=np.bool)
+
+        for i in range(n_features):
+            if np_version < (1, 8):
+                # in1d is not supported for object datatype in np < 1.8
+                valid_mask = np.ones_like(X[:, i], dtype=np.bool)
+                found_classes = set(np.unique(X[:, i]))
+                valid_classes = set(self.label_encoders_[i].classes_)
+                invalid_classes = found_classes - valid_classes
+
+                for item in invalid_classes:
+                    mask = X[:, i] == item
+                    np.logical_not(mask, mask)
+                    np.logical_and(valid_mask, mask, valid_mask)
+
+            else:
+                valid_mask = np.in1d(X[:, i], self.label_encoders_[i].classes_)
+
+            if not np.all(valid_mask):
+                if self.handle_unknown == 'error':
+                    if np_version < (1, 8):
+                        valid_classes = set(self.label_encoders_[i].classes_)
+                        diff = set(X[:, i]) - valid_classes
+                        diff = list(diff)
+                    else:
+                        diff = np.setdiff1d(X[:, i],
+                                            self.label_encoders_[i].classes_)
+                    msg = 'Unknown feature(s) %s in column %d' % (diff, i)
+                    raise ValueError(msg)
+                else:
+                    # Set the problematic rows to an acceptable value and
+                    # continue `The rows are marked `X_mask` and will be
+                    # removed later.
+                    X_mask[:, i] = valid_mask
+                    X[:, i][~valid_mask] = self.label_encoders_[i].classes_[0]
+            X_int[:, i] = self.label_encoders_[i].transform(X[:, i])
+
+        mask = X_mask.ravel()
+        n_values = [le.classes_.shape[0] for le in self.label_encoders_]
+        n_values = np.hstack([[0], n_values])
+        indices = np.cumsum(n_values)
+        self.feature_indices_ = indices
+
+        column_indices = (X_int + indices[:-1]).ravel()[mask]
+        row_indices = np.repeat(np.arange(n_samples, dtype=np.int32),
+                                n_features)[mask]
+        data = np.ones(n_samples * n_features)[mask]
+
+        out = sparse.coo_matrix((data, (row_indices, column_indices)),
+                                shape=(n_samples, indices[-1]),
+                                dtype=self.dtype).tocsr()
+
+        return out if self.sparse else out.toarray()
diff --git a/sklearn/preprocessing/tests/test_data.py b/sklearn/preprocessing/tests/test_data.py
index af7f28f8162c6..8c738d5b915b8 100644
--- a/sklearn/preprocessing/tests/test_data.py
+++ b/sklearn/preprocessing/tests/test_data.py
@@ -6,6 +6,7 @@
 from __future__ import division
 
 import warnings
+import re
 import numpy as np
 import numpy.linalg as la
 from scipy import sparse
@@ -31,13 +32,14 @@
 from sklearn.utils.testing import skip_if_32bit
 
 from sklearn.utils.sparsefuncs import mean_variance_axis
-from sklearn.preprocessing.data import _transform_selected
+from sklearn.preprocessing.data import _apply_selected
 from sklearn.preprocessing.data import _handle_zeros_in_scale
 from sklearn.preprocessing.data import Binarizer
 from sklearn.preprocessing.data import KernelCenterer
 from sklearn.preprocessing.data import Normalizer
 from sklearn.preprocessing.data import normalize
 from sklearn.preprocessing.data import OneHotEncoder
+from sklearn.preprocessing.data import CategoricalEncoder
 from sklearn.preprocessing.data import StandardScaler
 from sklearn.preprocessing.data import scale
 from sklearn.preprocessing.data import MinMaxScaler
@@ -1851,29 +1853,29 @@ def test_one_hot_encoder_dense():
                                  [1., 0., 1., 0., 1.]]))
 
 
-def _check_transform_selected(X, X_expected, sel):
+def _check_apply_selected(X, X_expected, sel):
     for M in (X, sparse.csr_matrix(X)):
-        Xtr = _transform_selected(M, Binarizer().transform, sel)
+        Xtr = _apply_selected(M, Binarizer().transform, sel)
         assert_array_equal(toarray(Xtr), X_expected)
 
 
-def test_transform_selected():
-    X = [[3, 2, 1], [0, 1, 1]]
+def test_apply_selected():
+    X = np.array([[3, 2, 1], [0, 1, 1]])
 
     X_expected = [[1, 2, 1], [0, 1, 1]]
-    _check_transform_selected(X, X_expected, [0])
-    _check_transform_selected(X, X_expected, [True, False, False])
+    _check_apply_selected(X, X_expected, [0])
+    _check_apply_selected(X, X_expected, [True, False, False])
 
     X_expected = [[1, 1, 1], [0, 1, 1]]
-    _check_transform_selected(X, X_expected, [0, 1, 2])
-    _check_transform_selected(X, X_expected, [True, True, True])
-    _check_transform_selected(X, X_expected, "all")
+    _check_apply_selected(X, X_expected, [0, 1, 2])
+    _check_apply_selected(X, X_expected, [True, True, True])
+    _check_apply_selected(X, X_expected, "all")
 
-    _check_transform_selected(X, X, [])
-    _check_transform_selected(X, X, [False, False, False])
+    _check_apply_selected(X, X, [])
+    _check_apply_selected(X, X, [False, False, False])
 
 
-def test_transform_selected_copy_arg():
+def test_apply_selected_copy_arg():
     # transformer that alters X
     def _mutating_transformer(X):
         X[0, 0] = X[0, 0] + 1
@@ -1883,7 +1885,7 @@ def _mutating_transformer(X):
     expected_Xtr = [[2, 2], [3, 4]]
 
     X = original_X.copy()
-    Xtr = _transform_selected(X, _mutating_transformer, copy=True,
+    Xtr = _apply_selected(X, _mutating_transformer, copy=True,
                               selected='all')
 
     assert_array_equal(toarray(X), toarray(original_X))
@@ -1952,6 +1954,57 @@ def test_one_hot_encoder_unknown_transform():
     assert_raises(ValueError, oh.transform, y)
 
 
+def check_categorical(X, cat_mask):
+    cat_idx = np.where(cat_mask)
+
+    enc = CategoricalEncoder(categorical_features=cat_mask)
+    Xtr1 = enc.fit_transform(X)
+
+    enc = CategoricalEncoder(categorical_features=cat_idx)
+    Xtr2 = enc.fit_transform(X)
+
+    enc = CategoricalEncoder(categorical_features=cat_mask, sparse=False)
+    Xtr3 = enc.fit_transform(X)
+
+    assert_allclose(Xtr1.toarray(), Xtr2.toarray())
+    assert_allclose(Xtr1.toarray(), Xtr3)
+
+    assert sparse.issparse(Xtr1)
+    assert sparse.issparse(Xtr2)
+    return Xtr1.toarray()
+
+
+def test_categorical_encoder():
+    X = [['abc', 1, 55], ['def', 2, 55]]
+
+    Xtr = check_categorical(X, [True, False, False])
+    assert_allclose(Xtr, [[1, 0, 1, 55], [0, 1, 2, 55]])
+
+    Xtr = check_categorical(X, [True, True, False])
+    assert_allclose(Xtr, [[1, 0, 1, 0,  55], [0, 1, 0, 1, 55]])
+
+    Xtr = CategoricalEncoder().fit_transform(X)
+    assert_allclose(Xtr.toarray(), [[1, 0, 1, 0,  1], [0, 1, 0, 1, 1]])
+
+
+def test_categorical_encoder_errors():
+
+    enc = CategoricalEncoder()
+    X = [[1, 2, 3], [4, 5, 6]]
+    enc.fit(X)
+
+    X[0][0] = -1
+    msg = re.escape('Unknown feature(s) [-1] in column 0')
+    assert_raises_regex(ValueError, msg, enc.transform, X)
+
+    enc = CategoricalEncoder(handle_unknown='ignore')
+    X = [[1, 2, 3], [4, 5, 6]]
+    enc.fit(X)
+    X[0][0] = -1
+    Xtr = enc.transform(X)
+    assert_allclose(Xtr.toarray(), [[0, 0, 1, 0, 1, 0], [0, 1, 0, 1, 0, 1]])
+
+
 def test_fit_cold_start():
     X = iris.data
     X_2d = X[:, :2]
diff --git a/sklearn/utils/testing.py b/sklearn/utils/testing.py
index 20731a9458885..10df72aebc0f6 100644
--- a/sklearn/utils/testing.py
+++ b/sklearn/utils/testing.py
@@ -519,7 +519,7 @@ def uninstall_mldata_mock():
              'LabelBinarizer', 'LabelEncoder',
              'MultiLabelBinarizer', 'TfidfTransformer',
              'TfidfVectorizer', 'IsotonicRegression',
-             'OneHotEncoder', 'RandomTreesEmbedding',
+             'OneHotEncoder', 'RandomTreesEmbedding', 'CategoricalEncoder',
              'FeatureHasher', 'DummyClassifier', 'DummyRegressor',
              'TruncatedSVD', 'PolynomialFeatures',
              'GaussianRandomProjectionHash', 'HashingVectorizer',


Code review comments:
diff_hunk: @@ -1197,7 +1197,7 @@ See the :ref:`metrics` section of the user guide for further details.
    preprocessing.MaxAbsScaler
    preprocessing.MinMaxScaler
    preprocessing.Normalizer
-   preprocessing.OneHotEncoder
path: doc/modules/classes.rst
text: @user1:
This indicated deprecation of the old class? I'm not sure we want to do that yet.

@author:
yeah, for now just rebased old pr. See my to do list in top post. I would for now just leave the OneHotEncoder as is. We can always decide to deprecate it later if we want

@user1:
Looked at the code before I looked at your description. I think your description is a good summary.

diff_hunk: @@ -2440,3 +2446,189 @@ def quantile_transform(X, axis=0, n_quantiles=1000,
     else:
         raise ValueError("axis should be either equal to 0 or 1. Got"
                          " axis={}".format(axis))
+
+
+class CategoricalEncoder(BaseEstimator, TransformerMixin):
+    """Encode categorical features using a one-hot aka one-of-K scheme.
+
+    The input to this transformer should be a matrix of integers or strings,
+    denoting the values taken on by categorical (discrete) features. The
+    output will be a sparse matrix where each column corresponds to one
+    possible value of one feature.
+
+    This encoding is needed for feeding categorical data to many scikit-learn
+    estimators, notably linear models and SVMs with the standard kernels.
+    Read more in the :ref:`User Guide <preprocessing_categorical_features>`.
+
+    Parameters
+    ----------
+    classes : 'auto', 2D array of ints or strings or both.
path: sklearn/preprocessing/data.py
text: @user1:
not 2d. A list of lists of values.

diff_hunk: @@ -2440,3 +2446,189 @@ def quantile_transform(X, axis=0, n_quantiles=1000,
     else:
         raise ValueError("axis should be either equal to 0 or 1. Got"
                          " axis={}".format(axis))
+
+
+class CategoricalEncoder(BaseEstimator, TransformerMixin):
+    """Encode categorical features using a one-hot aka one-of-K scheme.
+
+    The input to this transformer should be a matrix of integers or strings,
+    denoting the values taken on by categorical (discrete) features. The
+    output will be a sparse matrix where each column corresponds to one
+    possible value of one feature.
+
+    This encoding is needed for feeding categorical data to many scikit-learn
+    estimators, notably linear models and SVMs with the standard kernels.
+    Read more in the :ref:`User Guide <preprocessing_categorical_features>`.
+
+    Parameters
+    ----------
+    classes : 'auto', 2D array of ints or strings or both.
+        Values per feature.
+
+        - 'auto' : Determine classes automatically from the training data.
+        - array: ``classes[i]`` holds the classes expected in the ith column.
path: sklearn/preprocessing/data.py
text: @user1:
Does this specify the *order* of these values in the output, or just the set? Must be clear.

diff_hunk: @@ -2440,3 +2446,189 @@ def quantile_transform(X, axis=0, n_quantiles=1000,
     else:
         raise ValueError("axis should be either equal to 0 or 1. Got"
                          " axis={}".format(axis))
+
+
+class CategoricalEncoder(BaseEstimator, TransformerMixin):
+    """Encode categorical features using a one-hot aka one-of-K scheme.
+
+    The input to this transformer should be a matrix of integers or strings,
+    denoting the values taken on by categorical (discrete) features. The
+    output will be a sparse matrix where each column corresponds to one
+    possible value of one feature.
+
+    This encoding is needed for feeding categorical data to many scikit-learn
+    estimators, notably linear models and SVMs with the standard kernels.
+    Read more in the :ref:`User Guide <preprocessing_categorical_features>`.
+
+    Parameters
+    ----------
+    classes : 'auto', 2D array of ints or strings or both.
+        Values per feature.
+
+        - 'auto' : Determine classes automatically from the training data.
+        - array: ``classes[i]`` holds the classes expected in the ith column.
+
+    categorical_features : 'all' or array of indices or mask
path: sklearn/preprocessing/data.py
text: @user1:
I'd support getting rid of this as unnecesarry complexity.