bug: Nested Pipeline is not fitted (sklearn 1.8)

EDIT: The specific issue described here is actually caused by skrub, and solved in https://github.com/skrub-data/skrub/pull/1813.
However, this poses a more general question on how easy it is to abide by sklearn's conventions: skip directly to [this comment](https://github.com/scikit-learn/scikit-learn/issues/32910#issuecomment-3665534057) for that discussion.

---

### Describe the bug

With sklearn==1.8.0 and skrub==0.6.0, the following script fails:

```python
# test_failure.py

from sklearn.linear_model import Ridge
from sklearn.compose import make_column_transformer
from sklearn.pipeline import make_pipeline
from skrub import DatetimeEncoder, ToDatetime, DropCols
from skrub.datasets import fetch_employee_salaries

datasets = fetch_employee_salaries()

X, y = datasets.X, datasets.y

date_encoder = make_pipeline(
    ToDatetime(),
    DatetimeEncoder(),
    DropCols("date_first_hired_year"),
)

preprocessing = make_column_transformer(
    (date_encoder, "date_first_hired"),
)

model = make_pipeline(preprocessing, Ridge())

model.fit(X, y)
model.predict(X)
```

with stacktrace:
<details>
<summary>Stacktrace</summary>

```
/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/scipy/_lib/_util.py:1233: LinAlgWarning: Ill-conditioned matrix (rcond=1.36439e-16): result may not be accurate.
  return f(*arrays, *other_args, **kwargs)
Traceback (most recent call last):
  File "/home/auguste/Desktop/work/skore/examples/use_cases/test_failure.py", line 24, in <module>
    model.predict(X)
    ~~~~~~~~~~~~~^^^
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/sklearn/pipeline.py", line 745, in predict
    Xt = transform.transform(Xt)
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/sklearn/utils/_set_output.py", line 316, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/sklearn/compose/_column_transformer.py", line 1099, in transform
    Xs = self._call_func_on_transformers(
        X,
    ...<3 lines>...
        routed_params=routed_params,
    )
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/sklearn/compose/_column_transformer.py", line 901, in _call_func_on_transformers
    return Parallel(n_jobs=self.n_jobs)(jobs)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/sklearn/utils/parallel.py", line 91, in __call__
    return super().__call__(iterable_with_config_and_warning_filters)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/joblib/parallel.py", line 1986, in __call__
    return output if self.return_generator else list(output)
                                                ~~~~^^^^^^^^
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/joblib/parallel.py", line 1914, in _get_sequential_output
    res = func(*args, **kwargs)
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/sklearn/utils/parallel.py", line 184, in __call__
    return self.function(*args, **kwargs)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/sklearn/pipeline.py", line 1464, in _transform_one
    res = transformer.transform(X, **params.transform)
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/sklearn/pipeline.py", line 1035, in transform
    check_is_fitted(self)
    ~~~~~~~~~~~~~~~^^^^^^
  File "/home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/sklearn/utils/validation.py", line 1705, in check_is_fitted
    raise NotFittedError(msg % {"name": type(estimator).__name__})
sklearn.exceptions.NotFittedError: This Pipeline instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.
```

</details>

Indeed, if we look at the model after `.fit()` it seems some internal steps are not fitted:
```python
>>> model
Pipeline(steps=[('columntransformer',
                 ColumnTransformer(transformers=[('pipeline',
                                                  Pipeline(steps=[('todatetime',
                                                                   ToDatetime()),
                                                                  ('datetimeencoder',
                                                                   DatetimeEncoder()),
                                                                  ('dropcols',
                                                                   DropCols(cols='date_first_hired_year'))]),
                                                  'date_first_hired')])),
                ('ridge', Ridge())])
>>> model.__sklearn_is_fitted__()
True
>>> model.steps[0][1]
ColumnTransformer(transformers=[('pipeline',
                                 Pipeline(steps=[('todatetime', ToDatetime()),
                                                 ('datetimeencoder',
                                                  DatetimeEncoder()),
                                                 ('dropcols',
                                                  DropCols(cols='date_first_hired_year'))]),
                                 'date_first_hired')])
>>> model.steps[0][1].transformers[0][1].__sklearn_is_fitted__()
False
```

---

I used the debugger to dig a bit more inside `model.fit`, and I ended up in `sklearn.compose._column_transformer.py` at line 892.

<https://github.com/scikit-learn/scikit-learn/blob/1.8.0/sklearn/compose/_column_transformer.py#L892>

Notice that at this line, `trans` is *cloned* if `fitted` is False. If it's a clone of the transformer that gets fitted, I guess it makes sense that the original is not fitted?

Indeed the state when I reach this line is
```python
>>> self
ColumnTransformer(transformers=[('pipeline',
                                 Pipeline(steps=[('todatetime', ToDatetime()),
                                                 ('datetimeencoder',
                                                  DatetimeEncoder()),
                                                 ('dropcols',
                                                  DropCols(cols='date_first_hired_year'))]),
                                 'date_first_hired')])
>>> trans
Pipeline(steps=[('todatetime', ToDatetime()),
                ('datetimeencoder', DatetimeEncoder()),
                ('dropcols', DropCols(cols='date_first_hired_year'))])
>>> fitted
False
>>> func
<function _fit_transform_one at 0x7f0997d335e0>
```

---

However, if you replace the datetime pipeline with a scaling (skrub-less) pipeline, there is no error:
```python
# test_success.py

from sklearn.linear_model import Ridge
from sklearn.compose import make_column_transformer
from sklearn.pipeline import make_pipeline
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler

X, y = make_classification()

scaling = make_pipeline(StandardScaler())

preprocessing = make_column_transformer((scaling, [1]))

model = make_pipeline(preprocessing, Ridge())

model.fit(X, y)
model.predict(X)
```

Likewise, the error does not happen with sklearn 1.7.

For what it's worth, this might be linked to https://github.com/scikit-learn/scikit-learn/issues/29983 and https://github.com/scikit-learn/scikit-learn/issues/8157

### Versions

```shell
System:
    python: 3.14.0 (main, Nov 19 2025, 22:48:15) [Clang 21.1.4 ]
executable: /home/auguste/Desktop/work/skore/.venv/bin/python3
   machine: Linux-6.12.59-x86_64-with-glibc2.40

Python dependencies:
      sklearn: 1.8.0
          pip: None
   setuptools: 80.9.0
        numpy: 2.3.5
        scipy: 1.16.3
       Cython: None
       pandas: 2.3.3
   matplotlib: 3.10.8
       joblib: 1.5.3
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 12
         prefix: libscipy_openblas
       filepath: /home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/numpy.libs/libscipy_openblas64_-fdde5778.so
        version: 0.3.30
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 12
         prefix: libscipy_openblas
       filepath: /home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/scipy.libs/libscipy_openblas-b75cc656.so
        version: 0.3.29.dev
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 12
         prefix: libgomp
       filepath: /home/auguste/Desktop/work/skore/.venv/lib/python3.14/site-packages/scikit_learn.libs/libgomp-e985bcbb.so.1.0.0
        version: None
```

COMMENT:
You gave the answer that I was hoping for, but also dreading. From what you say about making a scikit-learn compatible estimator and what I've seen in cuml -> it is too hard to achieve this goal. I didn't comment before Christmas because I thought I should think about how we could solve this before replying. Now it is already the next year and I still have no great idea for a concrete thing to do that improves things. But so much time has passed that I should comment anyway.

Improving the documentation is clearly something to do. The hard part IMHO is how to do it so that it becomes easier to find the important information. For example, I think we should aim to make the relevant bits of documentation _shorter_, not longer as part of this work. Why shorter? Because it means it is easier to find stuff and because trying to make it shorter will counter our natural urge to "just add more information". But the details of how to do this are still unclear to me.

If we do revamp the docs and guides we should also make a blog post on the scikit-learn blog about the fact that this work has happened. Ideally repeating some of the information, because repetition is the way to get people to remember things.

If you have thoughts or want to try something out (and have the time) I think it would be good to make a PR with a proposal. Maybe too much thinking up front isn't actually all that useful?!

Labels: Bug, Needs Triage