Repository: scikit-learn/scikit-learn
Pull request: FIX Adds more informative error message for OHE
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes https://github.com/scikit-learn/scikit-learn/issues/26515


#### What does this implement/fix? Explain your changes.
This PR adds a more informative error message to OHE when `sparse_output=True` and there is pandas output.


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


Problem statement:
Improve error message with pandas output and sparse data
Currently, we raise an error:

```
ValueError: Pandas output does not support sparse data.
```

when a transformer does output sparse data and we try to wrap it inside a dataframe due to `set_output(transform="pandas")`. I assume that at this point, we could be extra nice and check if the transformer has a `sparse_output` keyword and advice changing it to `False`. We can also document that doing this change would be memory inefficient (it might blow up your RAM) but useful when it comes to data introspection.

What do you think about this proposal @thomasjpfan?

Hints from issue discussion:
I am +1 with adding a more informative error message for transformers with `sparse_output`.
Hi @thomasjpfan ,

I am new to open source and would love to work on this issue with your help and guidance.
@sqali You should start by going through the contribution guideline: https://scikit-learn.org/dev/developers/contributing.html

Regarding the code, you need to modify the error raised and change the unit tests that might break due to the change.

Commit message: TST Adjust comment test

Patch to review:
diff --git a/doc/whats_new/v1.4.rst b/doc/whats_new/v1.4.rst
index 1f66619ae2219..1bdce6df4d6c7 100644
--- a/doc/whats_new/v1.4.rst
+++ b/doc/whats_new/v1.4.rst
@@ -150,6 +150,13 @@ Changelog
   metric parameter `p` is in the range `0 < p < 1`, regardless of the `dtype` of `X`.
   :pr:`26760` by :user:`Shreesha Kumar Bhat <Shreesha3112>`.
 
+:mod:`sklearn.preprocessing`
+............................
+
+- |Fix| :class:`preprocessing.OneHotEncoder` shows a more informative error message
+  when `sparse_output=True` and the output is configured to be pandas.
+  :pr:`26931` by `Thomas Fan`_.
+
 :mod:`sklearn.tree`
 ...................
 
diff --git a/sklearn/preprocessing/_encoders.py b/sklearn/preprocessing/_encoders.py
index febc2523845ad..ada860e637a1a 100644
--- a/sklearn/preprocessing/_encoders.py
+++ b/sklearn/preprocessing/_encoders.py
@@ -14,6 +14,7 @@
 from ..utils._encode import _check_unknown, _encode, _get_counts, _unique
 from ..utils._mask import _get_mask
 from ..utils._param_validation import Hidden, Interval, RealNotInt, StrOptions
+from ..utils._set_output import _get_output_config
 from ..utils.validation import _check_feature_names_in, check_is_fitted
 
 __all__ = ["OneHotEncoder", "OrdinalEncoder"]
@@ -1008,6 +1009,14 @@ def transform(self, X):
             returned.
         """
         check_is_fitted(self)
+        transform_output = _get_output_config("transform", estimator=self)["dense"]
+        if transform_output == "pandas" and self.sparse_output:
+            raise ValueError(
+                "Pandas output does not support sparse data. Set"
+                " sparse_output=False to output pandas DataFrames or disable pandas"
+                " output."
+            )
+
         # validation of X happens in _check_X called by _transform
         warn_on_unknown = self.drop is not None and self.handle_unknown in {
             "ignore",
diff --git a/sklearn/preprocessing/tests/test_encoders.py b/sklearn/preprocessing/tests/test_encoders.py
index ca809dd513cf3..88134eb3137c9 100644
--- a/sklearn/preprocessing/tests/test_encoders.py
+++ b/sklearn/preprocessing/tests/test_encoders.py
@@ -1588,6 +1588,26 @@ def test_ohe_drop_first_explicit_categories(handle_unknown):
     assert_allclose(X_trans, X_expected)
 
 
+def test_ohe_more_informative_error_message():
+    """Raise informative error message when pandas output and sparse_output=True."""
+    pd = pytest.importorskip("pandas")
+    df = pd.DataFrame({"a": [1, 2, 3], "b": ["z", "b", "b"]}, columns=["a", "b"])
+
+    ohe = OneHotEncoder(sparse_output=True)
+    ohe.set_output(transform="pandas")
+
+    msg = (
+        "Pandas output is not support when sparse_output=True. Set "
+        "sparse_output=False to output pandas DataFrames or disable pandas output."
+    )
+    with pytest.raises(ValueError, match=msg):
+        ohe.fit_transform(df)
+
+    ohe.fit(df)
+    with pytest.raises(ValueError, match=msg):
+        ohe.transform(df)
+
+
 def test_ordinal_encoder_passthrough_missing_values_float_errors_dtype():
     """Test ordinal encoder with nan passthrough fails when dtype=np.int32."""
 
diff --git a/sklearn/utils/estimator_checks.py b/sklearn/utils/estimator_checks.py
index 128c1ffc35070..53ae056b4d2f7 100644
--- a/sklearn/utils/estimator_checks.py
+++ b/sklearn/utils/estimator_checks.py
@@ -4587,7 +4587,7 @@ def check_set_output_transform_pandas(name, transformer_orig):
         outputs_pandas = _output_from_fit_transform(transformer_pandas, name, X, df, y)
     except ValueError as e:
         # transformer does not support sparse data
-        assert str(e) == "Pandas output does not support sparse data.", e
+        assert "Pandas output does not support sparse data." in str(e), e
         return
 
     for case in outputs_default:
@@ -4633,7 +4633,7 @@ def check_global_output_transform_pandas(name, transformer_orig):
             )
     except ValueError as e:
         # transformer does not support sparse data
-        assert str(e) == "Pandas output does not support sparse data.", e
+        assert "Pandas output does not support sparse data." in str(e), e
         return
 
     for case in outputs_default:


Code review comments:
diff_hunk: @@ -1008,6 +1009,14 @@ def transform(self, X):
             returned.
         """
         check_is_fitted(self)
+        transform_output = _get_output_config("transform", estimator=self)["dense"]
+        if transform_output == "pandas" and self.sparse_output:
+            raise ValueError(
+                "Pandas output does not support sparse data. Set"
+                " sparse_output=False to output pandas DataFrames or disable pandas"
+                " output."
path: sklearn/preprocessing/_encoders.py
text: @user1:
```suggestion
                " output via `ohe.set_output(transform="default")."
```