Repository: scikit-learn/scikit-learn
Pull request: FIX Ensure dtype of categories is `object` for strings in `OneHotEncoder`
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Closes #25171

#### What does this implement/fix? Explain your changes.
This makes it so that the `enc.categories_` attribute of `OneHotEncoder` is contains an array of dtype `object` when using predefined categories that are strings. This makes it consistent with the dtype of `enc.categories_` when the categories are determine during `fit`. In general to compare a sequence of bytes to a string you need to assume an encoding, otherwise you can't really compare them. But I don't understand enough about the numpy type system to know if it would take care of this already? Ideas an input welcome.

Problem statement:
OneHotEncoder cuts predefined classes
### Describe the bug

When having predefined categories for the OneHotEncoder the categories get cut off. This lead to an error when trying to transform samples with the categories present....

### Steps/Code to Reproduce

```python
from sklearn.preprocessing import OneHotEncoder
import numpy as np
from sklearn.preprocessing import OneHotEncoder

# I know all my categories in advance and there will be no other
# Using only one sample to call fit since I need to call at some point

list_of_bounds = [["as","mmas","eas","ras","acs"], ["1", "2"]]

one_sample = ["as", "1"]

o_h_enc = OneHotEncoder(categories=list_of_bounds)
o_h_enc.fit(np.array(one_sample).reshape(1, -1) )


print(o_h_enc.categories_)
# [array(['as', 'mm', 'ea', 'ra', 'ac'], dtype='<U2'), array(['1', '2'], dtype='<U2')]
# The first category should be dtype object and contain the full names i.e.
# [array(["as","mmas","eas","ras","acs"], dtype='object'), array(['1', '2'], dtype='<U2')]

o_h_enc.transform(np.array(["mmas", "1"]).reshape((1, -1)))
```

### Expected Results

[array(["as","mmas","eas","ras","acs"], dtype='object'), array(['1', '2'], dtype='<U2')]


### Actual Results

```# [array(['as', 'mm', 'ea', 'ra', 'ac'], dtype='<U2'), array(['1', '2'], dtype='<U2')]
```

### Versions

```shell
Python dependencies:
      sklearn: 1.2.0
          pip: 21.1.2
   setuptools: 57.0.0
        numpy: 1.23.4
        scipy: 1.9.3
       Cython: None
       pandas: None
   matplotlib: 3.6.2
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True
None
```

Hints from issue discussion:
I think this happens because we convert the category for each feature to the dtype of the input ([on this line](https://github.com/scikit-learn/scikit-learn/blob/754bd5245aa46b89a1d686a3326c2b853012ff3e/sklearn/preprocessing/_encoders.py#L100)).

In this case `<U2` is used because the input is a two character string. I'm not sure if it should be of type `object`, though. I would have guessed that the dtype should be `<U4`, a four character string as that is the dtype of the known category.

However, when you don't provide categories to the constructor, then the resulting dtype is `object`. So maybe that is the correct dtype.

Commit message: Add explicit test for mixed string-bytes categories

Patch to review:
diff --git a/sklearn/preprocessing/_encoders.py b/sklearn/preprocessing/_encoders.py
index f7f82c3b728a7..5f637a8e76876 100644
--- a/sklearn/preprocessing/_encoders.py
+++ b/sklearn/preprocessing/_encoders.py
@@ -97,7 +97,15 @@ def _fit(
                 else:
                     cats = result
             else:
-                cats = np.array(self.categories[i], dtype=Xi.dtype)
+                if np.issubdtype(Xi.dtype, np.str_):
+                    # Always convert string categories to objects to avoid
+                    # unexpected string truncation for longer category labels
+                    # passed in the constructor.
+                    Xi_dtype = object
+                else:
+                    Xi_dtype = Xi.dtype
+
+                cats = np.array(self.categories[i], dtype=Xi_dtype)
                 if Xi.dtype.kind not in "OUS":
                     sorted_cats = np.sort(cats)
                     error_msg = (
diff --git a/sklearn/preprocessing/tests/test_encoders.py b/sklearn/preprocessing/tests/test_encoders.py
index 6395bc28c7d69..7953fcd8865c0 100644
--- a/sklearn/preprocessing/tests/test_encoders.py
+++ b/sklearn/preprocessing/tests/test_encoders.py
@@ -1376,6 +1376,22 @@ def test_encoders_string_categories(input_dtype, category_dtype, array_type):
     assert_array_equal(X_trans, expected)
 
 
+def test_mixed_string_bytes_categoricals():
+    """Check that this mixture of predefined categries and X raises an error.
+
+    Categories defined as bytes can not easily be compared to data that is
+    a string.
+    """
+    # data as unicode
+    X = np.array([["b"], ["a"]], dtype="U")
+    # predefined categories as bytes
+    categories = [np.array(["b", "a"], dtype="S")]
+    ohe = OneHotEncoder(categories=categories, sparse_output=False)
+
+    with pytest.raises(ValueError, match="Found unknown categories"):
+        ohe.fit(X)
+
+
 @pytest.mark.parametrize("missing_value", [np.nan, None])
 def test_ohe_missing_values_get_feature_names(missing_value):
     # encoder with missing values with object dtypes
@@ -1939,3 +1955,20 @@ def test_ordinal_set_output():
 
     assert_allclose(X_pandas.to_numpy(), X_default)
     assert_array_equal(ord_pandas.get_feature_names_out(), X_pandas.columns)
+
+
+def test_predefined_categories_dtype():
+    """Check that the categories_ dtype is `object` for string categories
+
+    Regression test for gh-25171.
+    """
+    categories = [["as", "mmas", "eas", "ras", "acs"], ["1", "2"]]
+
+    enc = OneHotEncoder(categories=categories)
+
+    enc.fit([["as", "1"]])
+
+    assert len(categories) == len(enc.categories_)
+    for n, cat in enumerate(enc.categories_):
+        assert cat.dtype == object
+        assert np.array_equal(categories[n], cat)


Code review comments:
diff_hunk: @@ -1376,6 +1376,22 @@ def test_encoders_string_categories(input_dtype, category_dtype, array_type):
     assert_array_equal(X_trans, expected)
 
 
+def test_mixed_string_bytes_categoricals():
+    """Check that this mixture of predefined categries and X raises an error.
path: sklearn/preprocessing/tests/test_encoders.py
text: @user1:
```suggestion
    """Check that this mixture of predefined categories and X raises an error.
```

diff_hunk: @@ -1939,3 +1955,20 @@ def test_ordinal_set_output():
 
     assert_allclose(X_pandas.to_numpy(), X_default)
     assert_array_equal(ord_pandas.get_feature_names_out(), X_pandas.columns)
+
+
+def test_predefined_categories_dtype():
+    """Check that the categories_ dtype is `object` for string categories
+
+    Regression test for gh-25171.
+    """
+    categories = [["as", "mmas", "eas", "ras", "acs"], ["1", "2"]]
+
+    enc = OneHotEncoder(categories=categories)
+
+    enc.fit([["as", "1"]])
+
+    assert len(categories) == len(enc.categories_)
+    for n, cat in enumerate(enc.categories_):
+        assert cat.dtype == object
+        assert np.array_equal(categories[n], cat)
path: sklearn/preprocessing/tests/test_encoders.py
text: @user1:
Use `assert_array_equal` from `sklearn.utils._testing`

```suggestion
        assert_array_equal(categories[n], cat)
```

diff_hunk: @@ -1376,6 +1376,22 @@ def test_encoders_string_categories(input_dtype, category_dtype, array_type):
     assert_array_equal(X_trans, expected)
 
 
+def test_mixed_string_bytes_categoricals():
+    """Check that this mixture of predefined categries and X raises an error.
+
+    Categories defined as bytes can not easily be compared to data that is
+    a string.
+    """
+    # data as unicode
+    X = np.array([["b"], ["a"]], dtype="U")
+    # predefined categories as bytes
+    categories = [np.array(["b", "a"], dtype="S")]
+    ohe = OneHotEncoder(categories=categories, sparse_output=False)
+
+    with pytest.raises(ValueError, match="Found unknown categories"):
path: sklearn/preprocessing/tests/test_encoders.py
text: @user1:
Reading the error message match, I find it quite unhelpful.
Could we catch it earlier by looking at the mismatch data type and raise a more informative error message?

@author:
I've now switched to "Found incompatible types between values and predefined categories in column 0.". I don't think we could raise the error much earlier than where it is now.