bug: Nested Pipeline is not fitted (sklearn 1.8)

EDIT: The specific issue described here is actually caused by skrub, and solved in https://github.com/skrub-data/skrub/pull/1813.
However, this poses a more general question on how easy it is to abide by sklearn's conventions: skip directly to [this comment](https://github.com/scikit-learn/scikit-learn/issues/32910#issuecomment-3665534057) for that discussion.

---

### Describe the bug

With sklearn==1.8.0 and skrub==0.6.0, the following script fails:

```python
# test_failure.py

from sklearn.linear_model import Ridge
from sklearn.compose import make_column_transformer
from sklearn.pipeline import make_pipeline
from skrub import DatetimeEncoder, ToDatetime, DropCols
from skrub.datasets import fetch_employee_salaries

datasets = fetch_employee_salaries()

X, y = datasets.X, datasets.y

date_encoder = make_pipeline(
    ToDatetime(),
    DatetimeEncoder(),
    DropCols("date_first_hired_year"),
)

preprocessing = make_column_transformer(
    (date_encoder, "date_first_hired"),
)

model = make_pipeline(preprocessing, Ridge())

model.fit(X, y)
model.predict(X)
```

with stacktrace:
<details>
<summary>Stacktrace</summary>

```
/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/scipy/_lib/_util.py:1233: LinAlgWarning: Ill-conditioned matrix (rcond=1.36439e-16): result may not be accurate.
  return f(*arrays, *other_args, **kwargs)
Traceback (most recent call last):
  File "/home/auguste/Desktop/work/skore/examples/use_cases/test_failure.py", line 24, in <module>
    model.predict(X)
    ~~~~~~~~~~~~~^^^
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/sklearn/pipeline.py", line 745, in predict
    Xt = transform.transform(Xt)
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/sklearn/utils/_set_output.py", line 316, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/sklearn/compose/_column_transformer.py", line 1099, in transform
    Xs = self._call_func_on_transformers(
        X,
    ...<3 lines>...
        routed_params=routed_params,
    )
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/sklearn/compose/_column_transformer.py", line 901, in _call_func_on_transformers
    return Parallel(n_jobs=self.n_jobs)(jobs)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/sklearn/utils/parallel.py", line 91, in __call__
    return super().__call__(iterable_with_config_and_warning_filters)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/joblib/parallel.py", line 1986, in __call__
    return output if self.return_generator else list(output)
                                                ~~~~^^^^^^^^
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/joblib/parallel.py", line 1914, in _get_sequential_output
    res = func(*args, **kwargs)
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/sklearn/utils/parallel.py", line 184, in __call__
    return self.function(*args, **kwargs)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/sklearn/pipeline.py", line 1464, in _transform_one
    res = transformer.transform(X, **params.transform)
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/sklearn/pipeline.py", line 1035, in transform
    check_is_fitted(self)
    ~~~~~~~~~~~~~~~^^^^^^
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/sklearn/utils/validation.py", line 1705, in check_is_fitted
    raise NotFittedError(msg % {"name": type(estimator).__name__})
sklearn.exceptions.NotFittedError: This Pipeline instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.
```

</details>

Indeed, if we look at the model after `.fit()` it seems some internal steps are not fitted:
```python
>>> model
Pipeline(steps=[('columntransformer',
                 ColumnTransformer(transformers=[('pipeline',
                                                  Pipeline(steps=[('todatetime',
                                                                   ToDatetime()),
                                                                  ('datetimeencoder',
                                                                   DatetimeEncoder()),
                                                                  ('dropcols',
                                                                   DropCols(cols='date_first_hired_year'))]),
                                                  'date_first_hired')])),
                ('ridge', Ridge())])
>>> model.__sklearn_is_fitted__()
True
>>> model.steps[0][1]
ColumnTransformer(transformers=[('pipeline',
                                 Pipeline(steps=[('todatetime', ToDatetime()),
                                                 ('datetimeencoder',
                                                  DatetimeEncoder()),
                                                 ('dropcols',
                                                  DropCols(cols='date_first_hired_year'))]),
                                 'date_first_hired')])
>>> model.steps[0][1].transformers[0][1].__sklearn_is_fitted__()
False
```

---

I used the debugger to dig a bit more inside `model.fit`, and I ended up in `sklearn.compose._column_transformer.py` at line 892.

<https://github.com/scikit-learn/scikit-learn/blob/1.8.0/sklearn/compose/_column_transformer.py#L892>

Notice that at this line, `trans` is *cloned* if `fitted` is False. If it's a clone of the transformer that gets fitted, I guess it makes sense that the original is not fitted?

Indeed the state when I reach this line is
```python
>>> self
ColumnTransformer(transformers=[('pipeline',
                                 Pipeline(steps=[('todatetime', ToDatetime()),
                                                 ('datetimeencoder',
                                                  DatetimeEncoder()),
                                                 ('dropcols',
                                                  DropCols(cols='date_first_hired_year'))]),
                                 'date_first_hired')])
>>> trans
Pipeline(steps=[('todatetime', ToDatetime()),
                ('datetimeencoder', DatetimeEncoder()),
                ('dropcols', DropCols(cols='date_first_hired_year'))])
>>> fitted
False
>>> func
<function _fit_transform_one at 0x7f0997d335e0>
```

---

However, if you replace the datetime pipeline with a scaling (skrub-less) pipeline, there is no error:
```python
# test_success.py

from sklearn.linear_model import Ridge
from sklearn.compose import make_column_transformer
from sklearn.pipeline import make_pipeline
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler

X, y = make_classification()

scaling = make_pipeline(StandardScaler())

preprocessing = make_column_transformer((scaling, [1]))

model = make_pipeline(preprocessing, Ridge())

model.fit(X, y)
model.predict(X)
```

Likewise, the error does not happen with sklearn 1.7.

For what it's worth, this might be linked to https://github.com/scikit-learn/scikit-learn/issues/29983 and https://github.com/scikit-learn/scikit-learn/issues/8157

### Versions

```shell
System:
    python: 3.14.0 (main, Nov 19 2025, 22:48:15) [Clang 21.1.4 ]
executable: /home/auguste/Desktop/work/skore/.venv/bin/python3
   machine: Linux-6.12.59-x86_64-with-glibc2.40

Python dependencies:
      sklearn: 1.8.0
          pip: None
   setuptools: 80.9.0
        numpy: 2.3.5
        scipy: 1.16.3
       Cython: None
       pandas: 2.3.3
   matplotlib: 3.10.8
       joblib: 1.5.3
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 12
         prefix: libscipy_openblas
       filepath: /home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/numpy.libs/libscipy_openblas64_-fdde5778.so
        version: 0.3.30
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 12
         prefix: libscipy_openblas
       filepath: /home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/scipy.libs/libscipy_openblas-b75cc656.so
        version: 0.3.29.dev
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 12
         prefix: libgomp
       filepath: /home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/scikit_learn.libs/libgomp-e985bcbb.so.1.0.0
        version: None
```

COMMENT:
I think the immediate fix is something to do in skrub.

Coming back to the bigger question about how difficult it is to make a scikit-learn compatible estimator, so basically preventing this problem from happening in the first place.

Does `DropCols` pass the common estimator checks? Does skrub use them to "certify" the estimators that want to be scikit-learn estimators?

I agree that creating a conforming scikti-learn estimator is quite difficult - in the sense that you have to do a lot of things that are not obvious from the start and only later realise that you missed it. For example the case in this issue: it makes sense that the generic "is this fitted?" checker relies on there being a fitted attribute. But, as estimator implementor, it is not clear that you have to have a fitted attribute. Long story short, I think the common tests are the only hope in gaining confidence as estimator implementor that you got it right. This means there are two questions: are these common checks easy enough to discover and use? Do the current checks catch this problem?

If the answer is no, to either of these, there is work to be done in scikit-learn.

Labels: Bug, Needs Triage