Talip7/scikit-learn-issues-multilabel huggingface dataset:
Handle np.nan / missing values in SplineTransformer

I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.

COMMENT:
@ogrisel Thanks for the example.
For a fair comparison, could you change to `SimpleImputer(strategy="constant", fill_value=0, add_indicator=True)`?

> `plt.bar(np.arange(len(coef)), coef * impute_indicator_then_spline[:-1].transform(X).std(axis=0));`

What do you intend to plot here?

Labels: Enhancement
Handle np.nan / missing values in SplineTransformer

I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.

COMMENT:
> What do you intend to plot here?

This was a left over from an experiment to see if the conclusion would be visually different by plotting "feature effects" rather than raw coefs. I edited my comment to just plot the coefs and the conclusion stays the same.

> For a fair comparison, could you change to SimpleImputer(strategy="constant", fill_value=0, add_indicator=True)?

Let me do the constant alternative with a constant at 0 but I expect a similar problem (just with a different spline feature).

Labels: Enhancement
Handle np.nan / missing values in SplineTransformer

I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.

COMMENT:
We indeed get the same problem:

```python
from sklearn.compose import ColumnTransformer

impute_indicator_then_spline = make_pipeline(
    SimpleImputer(strategy="constant", fill_value=0, add_indicator=True),
    ColumnTransformer(
        [
            ("spline", SplineTransformer(n_knots=n_knots), [0]),
        ],
        remainder="passthrough",
    ),
    lr,
).fit(X, y)


coef = impute_indicator_then_spline[-1].coef_.ravel()
plt.bar(np.arange(len(coef)), coef);
```
![image](https://github.com/scikit-learn/scikit-learn/assets/89061/74ba95d4-77a5-43b7-a1f6-491c1b2b0b8d)

Labels: Enhancement
Handle np.nan / missing values in SplineTransformer

I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.

COMMENT:
@lorentzenchr WDYT?

I think it's very nice to have an inductive bias that decouples "missingness" from any other non-missing value in the output feature space.

We could also implement the "zero"/"ignore" strategy that would show a similar inductive bias as `OneHotEncoder(handle_missing="ignore")`. But then missingness representation would be more implicitly and uniformly spread over all output features + intercept.

Labels: Enhancement
Handle np.nan / missing values in SplineTransformer

I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.

COMMENT:
```python
def create_nan_handling_spline(n_knots):
    return make_pipeline(
        # Note that zero values in the feature matrix for linear models have a special meaning:
        # the coefficients have then no impact at all.
        SimpleImputer(strategy="constant", fill_value=0, add_indicator=True),
        SplineTransformer(n_knots=n_knots),
    )
```
~~This pretty much solves this issue. I think it is enough to have it documented somewhere, be it in the docstring or an example.~~

Edit: No, it does not.

Labels: Enhancement
Handle np.nan / missing values in SplineTransformer

I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.

COMMENT:
I'm not sure if I understand your point here @lorentzenchr 

The splines move the input into a very different space. The coefficient for what used to be 0, is now meaningful, cause it's not 0 anymore. But even if that was true, that would only hold for linear models. The other models can pick up signal from 0 and use that, so @ogrisel 's argument holds.

I read the whole thread with a fresh eye now, and I'm quite convinced that adding missing values here makes very much sense.

But I think there are two ways to go about adding "missing value support" here:

1. `SplineTransformer(handle_missing="indicator")` adds an indicator for missing value and returns 0 for all the splines if that's the case
2. `SplineTransformer(handle_missing="ignore")` returns 0 on all splines and we add a `MissingIndicator` in the pipeline which would be equivalent to the former implementation. Then we could document how this can be implemented by the user.

I personally don't know where I stand between the two options. But I think we need at least one of the two.

Labels: Enhancement
Handle np.nan / missing values in SplineTransformer

I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.

COMMENT:
> This pretty much solves this issue. I think it is enough to have it documented somewhere, be it in the docstring or an example.

No it does not: spline feature with index 0 would have a non-zero value when the original feature has a missing value: this is precisely the problem I want to avoid here.

Labels: Enhancement
Handle np.nan / missing values in SplineTransformer

I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.

COMMENT:
Ok, then let's add missing value support.

Labels: Enhancement
Handle np.nan / missing values in SplineTransformer

I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.

COMMENT:
> But I think there are two ways to go about adding "missing value support" here:
>
>   1. SplineTransformer(handle_missing="indicator") adds an indicator for missing value and returns 0 for all the splines if that's the case
>   2. SplineTransformer(handle_missing="ignore") returns 0 on all splines and we add a MissingIndicator in the pipeline which would be equivalent to the former implementation. Then we could document how this can be implemented by the user.
>
> I personally don't know where I stand between the two options. But I think we need at least one of the two.

I think we should definitely implement at least 1 (`handle_missing="indicator"`). because it makes it possible to explicitly model missingness as target-predictive conditional information.

That being said, make it is possible to not add those explicit features (via `handle_missing="ignore"`) could also be helpful to nudge the model towards not using the missingness pattern and avoid overfitting it when the modeler has prior knowledge that it should not be informative.

So +1 for implementing both in hindsight. Also, `handle_missing="ignore"` should be very easy to add once `handle_missing="indicator"` is implemented. @StefanieSenger feel free to only focus on `handle_missing="indicator"` in your current PR and we can defer the implementation of `handle_missing="ignore"` to a follow-up work.

Labels: Enhancement
Handle np.nan / missing values in SplineTransformer

I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.

COMMENT:
`handle_missing="zero"` has priority in my opinion.

Labels: Enhancement
Handle np.nan / missing values in SplineTransformer

I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.

COMMENT:
@lorentzenchr , I'm really not sure what `handle_missing="zero"` brings. That doesn't even do anything for the linear models. Unless there's something I'm missing in your analysis.

Labels: Enhancement