I'm not currently using the transformed `X` and `y` in the key.

I can see some benefits in using such a less clever version, though had hoped to allow estimators to provide a `_non_fit_params` to indicate those that shouldn't be used in the key.

I guess if we assume `Pipeline`s are not very long, doing 2 or 3 cache lookups, one for each component, is not much worse than doing 1 to get them all.