Repository: scikit-learn-contrib/category_encoders

Issue:
Combining with set_output can produce errors
## Expected Behavior
Calling set_output(transformer="pandas") should have no impact on a OneHotEncoder, since the outputs are already dataframes.

## Actual Behavior
The OneHotEncoder develops an inconsistent behavior, producing errors in the case of subsequent fit_transform calls.

## Steps to Reproduce the Problem

```
df = pd.DataFrame({"C1": ["a", "a"], "C2": ["c", "d"]})
ohe = OneHotEncoder().set_output(transform="pandas")
ohe.fit_transform(df.iloc[:1]) # commenting this line avoids the error
ohe.fit_transform(df.iloc[:2]) # this line produces an error
```

ValueError: Length mismatch: Expected axis has 3 elements, new values have 2 elements

## Specifications

pandas : 2.0.1
sklearn : 1.3.2
category_encoders : 2.6.3


Discussion:
I think I see what's happening. In `BaseEncoder.fit`, https://github.com/scikit-learn-contrib/category_encoders/blob/06e46db07ff15c362d7de5ad225bb0bffc245369/category_encoders/utils.py#L323-L324
we set `feature_names_out_` by looking at the encoder's transformed output (L324). But when sklearn is set to produce pandas output, it uses that attribute if available at L323. So it uses the old attribute. Depending on how sklearn picks its column names, we may be able to just delete the attribute somewhere earlier in `fit`. This probably affects all the encoders, too, not just OHE.

Update: I tried it out, and the error doesn't get thrown for most other encoders, because they don't change the number of columns; `BinaryEncoder` similarly fails.  But even for other encoders, making the second `fit_transform` on a dataframe with _new column names_ ends up using the old column names.  Adding just `self.feature_names_out_ = None` before L323 above fixes the problem in all cases, PR to come.



Patch:
diff --git a/category_encoders/utils.py b/category_encoders/utils.py
index 1671fc55..0db33552 100644
--- a/category_encoders/utils.py
+++ b/category_encoders/utils.py
@@ -320,6 +320,7 @@ def fit(self, X, y=None, **kwargs):
         self._fit(X, y, **kwargs)
 
         # for finding invariant columns transform without y (as is done on the test set)
+        self.feature_names_out_ = None  # Issue#437
         X_transformed = self.transform(X, override_return_df=True)
         self.feature_names_out_ = X_transformed.columns.tolist()
 
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 8d8db879..7cda0836 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -1,4 +1,5 @@
 sphinx
 sphinx_rtd_theme
 pytest
-numpydoc
\ No newline at end of file
+numpydoc
+packaging
\ No newline at end of file


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
index 3f4ead47..65799ca2 100644
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -1,5 +1,10 @@
 from unittest import TestCase  # or `from unittest import ...` if on Python 3.4+
-from category_encoders.utils import convert_input_vector, convert_inputs, get_categorical_cols
+import pytest
+from category_encoders.utils import convert_input_vector, convert_inputs, get_categorical_cols, BaseEncoder
+
+from sklearn.base import BaseEstimator, TransformerMixin
+from sklearn import __version__ as skl_version
+from packaging.version import Version
 import pandas as pd
 import numpy as np
 
@@ -120,3 +125,26 @@ def test_get_categorical_cols(self):
         self.assertEqual(get_categorical_cols(df.astype("object")), ["col"])
         self.assertEqual(get_categorical_cols(df.astype("category")), ["col"])
         self.assertEqual(get_categorical_cols(df.astype("string")), ["col"])
+
+
+class TestBaseEncoder(TestCase):
+    def setUp(self):
+        class DummyEncoder(BaseEncoder, BaseEstimator, TransformerMixin):
+            def _fit(self, X, y=None):
+                return self
+
+            def transform(self, X, y=None, override_return_df=False):
+                return X
+
+        self.encoder = DummyEncoder()
+
+    @pytest.mark.skipif(Version(skl_version) < Version('1.2'), reason="requires sklean > 1.2")
+    def test_sklearn_pandas_out_refit(self):
+        # Thanks to Issue#437
+        df = pd.DataFrame({"C1": ["a", "a"], "C2": ["c", "d"]})
+        self.encoder.set_output(transform="pandas")
+        self.encoder.fit_transform(df.iloc[:1])
+        out = self.encoder.fit_transform(
+                df.rename(columns={'C1': 'X1', 'C2': 'X2'})
+        )
+        self.assertTrue(list(out.columns) == ['X1', 'X2'])