Repository: scikit-learn/scikit-learn
Pull request: FIX Fixes unknown handling for str dtypes in OrdinalEncoder.transform
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes https://github.com/scikit-learn/scikit-learn/issues/19872

#### What does this implement/fix? Explain your changes.

When `Xi` is of dtype '<U1` and `self.categories_[i][0]` is 'AA', the following would collapse 'AA' into 'A':

```py
Xi[~valid_mask] = self.categories_[i][0]
```

This PR converts `Xi` into an object dtype when it sees that `self.categories_[i]` is an object dtype.


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


Problem statement:
Ordinal encoder is not able to raising error inspite of  the handle_unknown = 'use_encoded_value
#### Describe the bug
Using OrdinalEncoder(handle_unknown = 'use_encoded_value', unknown_value = -9)

I expected it to handle all the unknown values. But it seems to fail if we got a value which is lexicographically smaller than all the values OrdinalEncoder is fitted during the training.


#### Steps/Code to Reproduce

Example:
```python
import numpy as np
from sklearn.preprocessing import OrdinalEncoder
enc = OrdinalEncoder(handle_unknown='use_encoded_value', unknown_value=-9)
PP = [['AA','B']]
enc.fit(PP)

print(enc.transform(np.array(['AC','B']).reshape(1,-1)))    # Works as expected
print(enc.transform(np.array(['A','B']).reshape(1,-1)))   # raises value error
```

#### Expected Results

```python
print(enc.transform(np.array(['A','B']).reshape(1,-1)))  
```
This should output as `[[-9.  0.]]`.

#### Actual Results
```pytb
  KeyError                                  Traceback (most recent call last)
  /usr/local/lib/python3.7/dist-packages/sklearn/utils/_encode.py in _encode(values, uniques, check_unknown)
      177         try:
  --> 178             return _map_to_integer(values, uniques)
      179         except KeyError as e:
  
  6 frames
  /usr/local/lib/python3.7/dist-packages/sklearn/utils/_encode.py in _map_to_integer(values, uniques)
      122     table = _nandict({val: i for i, val in enumerate(uniques)})
  --> 123     return np.array([table[v] for v in values])
      124 
  
  /usr/local/lib/python3.7/dist-packages/sklearn/utils/_encode.py in <listcomp>(.0)
      122     table = _nandict({val: i for i, val in enumerate(uniques)})
  --> 123     return np.array([table[v] for v in values])
      124 
  
  /usr/local/lib/python3.7/dist-packages/sklearn/utils/_encode.py in __missing__(self, key)
      116             return self.nan_value
  --> 117         raise KeyError(key)
      118 
  
  KeyError: 'A'
  
  During handling of the above exception, another exception occurred:
  
  ValueError                                Traceback (most recent call last)
  <ipython-input-58-39a40b708985> in <module>()
        6 
        7 #print(enc.transform(np.array(['AC','B']).reshape(1,-1)))
  ----> 8 print(enc.transform(np.array(['A','B']).reshape(1,-1)))
  
  /usr/local/lib/python3.7/dist-packages/sklearn/preprocessing/_encoders.py in transform(self, X)
      785             Transformed input.
      786         """
  --> 787         X_int, X_mask = self._transform(X, handle_unknown=self.handle_unknown)
      788         X_trans = X_int.astype(self.dtype, copy=False)
      789 
  
  /usr/local/lib/python3.7/dist-packages/sklearn/preprocessing/_encoders.py in _transform(self, X, handle_unknown, force_all_finite)
      152             # already called above.
      153             X_int[:, i] = _encode(Xi, uniques=self.categories_[i],
  --> 154                                   check_unknown=False)
      155 
      156         return X_int, X_mask
  
  /usr/local/lib/python3.7/dist-packages/sklearn/utils/_encode.py in _encode(values, uniques, check_unknown)
      178             return _map_to_integer(values, uniques)
      179         except KeyError as e:
  --> 180             raise ValueError(f"y contains previously unseen labels: {str(e)}")
      181     else:
      182         if check_unknown:
  
  ValueError: y contains previously unseen labels: 'A'
```

#### Versions

System:
    python: 3.7.10 (default, Feb 20 2021, 21:17:23)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 19.3.1
   setuptools: 54.2.0
      sklearn: 0.24.1
        numpy: 1.19.5
        scipy: 1.4.1
       Cython: 0.29.22
       pandas: 1.1.5
   matplotlib: 3.2.2
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
Python 3.7.10 (default, Feb 20 2021, 21:17:23) 
[GCC 7.5.0]
NumPy 1.19.5
SciPy 1.4.1
Scikit-Learn 0.24.1
```
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-61-fdf473be56af> in <module>()
      6 import scipy; print("SciPy", scipy.__version__)
      7 import sklearn; print("Scikit-Learn", sklearn.__version__)
----> 8 import imblearn; print("Imbalanced-Learn", imblearn.__version__)

1 frames
/usr/local/lib/python3.7/dist-packages/imblearn/base.py in <module>()
     13 
     14 from sklearn.base import BaseEstimator
---> 15 from sklearn.externals import six
     16 from sklearn.preprocessing import label_binarize
     17 from sklearn.utils import check_X_y

ImportError: cannot import name 'six' from 'sklearn.externals' (/usr/local/lib/python3.7/dist-packages/sklearn/externals/__init__.py)


sklearn.externals  import six failed , but six is already imported using  import six  
```

Hints from issue discussion:
You are not using the same dtype between `fit` and `transform`. Passing a Python list will be converted to a numpy array of dtype `object` while you are passing a NumPy of string dtype at transform:

```python
import numpy as np
from sklearn.preprocessing import OrdinalEncoder
enc = OrdinalEncoder(handle_unknown='use_encoded_value', unknown_value=-9)
PP = [['AA','B']]
enc.fit(PP)

print(enc.transform(np.array(['AC','B'], dtype=object).reshape(1,-1)))
print(enc.transform(np.array(['A','B'], dtype=object).reshape(1,-1)))
```

Indeed, you can see that if you create the array, you have different dtype:

```py
In [2]: np.array(['A','B'])
Out[2]: array(['A', 'B'], dtype='<U1')

In [3]: np.array(['AC','B'])
Out[3]: array(['AC', 'B'], dtype='<U2')
```

I am not sure what is the root of the problem and how to solve it. I also not recall if we should support whether or not the string NumPy dtype. ping @rth @thomasjpfan 
It might be related to https://github.com/scikit-learn/scikit-learn/issues/19677, I am not sure

Commit message: DOC Adds whats new

Patch to review:
diff --git a/doc/whats_new/v1.0.rst b/doc/whats_new/v1.0.rst
index 23211cd3a95b1..c0480fa52456d 100644
--- a/doc/whats_new/v1.0.rst
+++ b/doc/whats_new/v1.0.rst
@@ -276,7 +276,7 @@ Changelog
   :pr:`18649` by `Leandro Hermida <hermidalc>` and
   `Rodion Martynov <marrodion>`.
 
-- |Fix| The `fit` method of the successive halving parameter search 
+- |Fix| The `fit` method of the successive halving parameter search
   (:class:`model_selection.HalvingGridSearchCV`, and
   :class:`model_selection.HalvingRandomSearchCV`) now correctly handles the
   `groups` parameter. :pr:`19847` by :user:`Xiaoyu Chai <xiaoyuchai>`.
@@ -347,6 +347,9 @@ Changelog
   supporting sparse matrix and raise the appropriate error message.
   :pr:`19879` by :user:`Guillaume Lemaitre <glemaitre>`.
 
+- |Fix| :meth:`preprocessing.OrdinalEncoder.transfrom` now correctly handles
+  unknown values for string dtypes. :pr:`19888` by `Thomas Fan`_.
+
 :mod:`sklearn.tree`
 ...................
 
diff --git a/sklearn/preprocessing/_encoders.py b/sklearn/preprocessing/_encoders.py
index cd05dc89bb75d..daad8b27c77f4 100644
--- a/sklearn/preprocessing/_encoders.py
+++ b/sklearn/preprocessing/_encoders.py
@@ -150,6 +150,10 @@ def _transform(self, X, handle_unknown='error', force_all_finite=True,
                     if (self.categories_[i].dtype.kind in ('U', 'S')
                             and self.categories_[i].itemsize > Xi.itemsize):
                         Xi = Xi.astype(self.categories_[i].dtype)
+                    elif (self.categories_[i].dtype.kind == 'O' and
+                            Xi.dtype.kind == 'U'):
+                        # categories are objects and Xi are numpy strings
+                        Xi = Xi.astype('O')
                     else:
                         Xi = Xi.copy()
 
diff --git a/sklearn/preprocessing/tests/test_encoders.py b/sklearn/preprocessing/tests/test_encoders.py
index 9f1e331f78fec..94e2c276dcd58 100644
--- a/sklearn/preprocessing/tests/test_encoders.py
+++ b/sklearn/preprocessing/tests/test_encoders.py
@@ -1132,3 +1132,23 @@ def test_ordinal_encoder_sparse():
     X_trans_sparse = sparse.csr_matrix(X_trans)
     with pytest.raises(TypeError, match=err_msg):
         encoder.inverse_transform(X_trans_sparse)
+
+
+@pytest.mark.parametrize("X_train", [
+    [['AA', 'B']],
+    np.array([['AA', 'B']], dtype='O'),
+    np.array([['AA', 'B']], dtype='U'),
+])
+@pytest.mark.parametrize("X_test", [
+    [['A', 'B']],
+    np.array([['A', 'B']], dtype='O'),
+    np.array([['A', 'B']], dtype='U'),
+])
+def test_ordinal_encoder_handle_unknown_string_dtypes(X_train, X_test):
+    """Checks that ordinal encoder transforms string dtypes. Non-regression
+    test for #19872."""
+    enc = OrdinalEncoder(handle_unknown='use_encoded_value', unknown_value=-9)
+    enc.fit(X_train)
+
+    X_trans = enc.transform(X_test)
+    assert_allclose(X_trans, [[-9, 0]])


Code review comments:
diff_hunk: @@ -150,6 +150,10 @@ def _transform(self, X, handle_unknown='error', force_all_finite=True,
                     if (self.categories_[i].dtype.kind in ('U', 'S')
                             and self.categories_[i].itemsize > Xi.itemsize):
                         Xi = Xi.astype(self.categories_[i].dtype)
+                    elif (self.categories_[i].dtype.kind == 'O' and
+                            Xi.dtype.kind == 'U'):
+                        # categories are objects and Xi are numpy strings
path: sklearn/preprocessing/_encoders.py
text: @user1:
You could maybe add that otherwise the string will be truncated