Talip7/scikit-learn-issues-multilabel huggingface dataset:
OneHotEncoder creates non unique column names, when values None (NoneType) and "None" (str) are both present

### Describe the bug

OneHotEncoder creates non unique column names when values None (NoneType) and "None" (str) are both present in the same column.

Such policy creates downstream problems (e.g. ColumnTransformer raising because of non unique column names).

It's not 100% clear to me what's the expectation of a transformer.
Should every transformer emit unique names for features?
Why does the ColumnTransformer raise if there are non-unique names?

### Steps/Code to Reproduce

Just one hot encoder.
```py
from sklearn.preprocessing import OneHotEncoder

X = [['None'], [None]]
t = OneHotEncoder().fit(X)
feature_names = t.get_feature_names_out()
# ['x0_None' 'x0_None']

assert len(feature_names) == len(set(feature_names))
```


With ColumnTransformer
```py
import pandas as pd
from sklearn.compose import make_column_selector, make_column_transformer
from sklearn.preprocessing import OneHotEncoder

t = make_column_transformer(
    (OneHotEncoder(), make_column_selector("x0")),
    verbose_feature_names_out=False,
)
t.fit(pd.DataFrame({"x0": [None, "None"]}))
print(t.get_feature_names_out(["x0"]))
```

There's a subtle bug in the ColumnTransformer as well. If `verbose_feature_names_out` is True, it will not check names and will emit non-unique names without raising.

### Expected Results

I'd expect different feature names being generated.

### Actual Results

First code snippet
```py
['x0_None' 'x0_None']
```

Second code snippet
```py
Traceback (most recent call last):
  File "src/ohe_bug.py", line 10, in <module>
    print(t.get_feature_names_out(["x0"]))
  File "/home/pilotuser/.local/lib/python3.8/site-packages/sklearn/compose/_column_transformer.py", line 518, in get_feature_names_out
    raise ValueError(
ValueError: Output feature names: ['x0_None'] are not unique. Please set verbose_feature_names_out=True to add prefixes to feature names
```

### Versions

```shell
System:
    python: 3.8.12 (default, Sep 28 2021, 19:23:30)  [GCC 8.3.0]
executable: /usr/local/bin/python
   machine: Linux-4.14.238-182.422.amzn2.x86_64-x86_64-with-glibc2.2.5

Python dependencies:
          pip: 21.2.4
   setuptools: 46.0.0
      sklearn: 1.0.2
        numpy: 1.18.5
        scipy: 1.5.4
       Cython: None
       pandas: 1.1.1
   matplotlib: 3.2.2
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```
```

COMMENT:
Indeed, the expected behaviour would be to get 2 different names there.
I could imagine that we could have the same bug with names reserved in Python (e.g. True vs "True").

It should be possible to detect the collision since the `repr` of `None` and `"None"` will differ.
However, I don't know if we should error or make some magic under the hood.

Labels: Bug, module:compose
OneHotEncoder creates non unique column names, when values None (NoneType) and "None" (str) are both present

### Describe the bug

OneHotEncoder creates non unique column names when values None (NoneType) and "None" (str) are both present in the same column.

Such policy creates downstream problems (e.g. ColumnTransformer raising because of non unique column names).

It's not 100% clear to me what's the expectation of a transformer.
Should every transformer emit unique names for features?
Why does the ColumnTransformer raise if there are non-unique names?

### Steps/Code to Reproduce

Just one hot encoder.
```py
from sklearn.preprocessing import OneHotEncoder

X = [['None'], [None]]
t = OneHotEncoder().fit(X)
feature_names = t.get_feature_names_out()
# ['x0_None' 'x0_None']

assert len(feature_names) == len(set(feature_names))
```


With ColumnTransformer
```py
import pandas as pd
from sklearn.compose import make_column_selector, make_column_transformer
from sklearn.preprocessing import OneHotEncoder

t = make_column_transformer(
    (OneHotEncoder(), make_column_selector("x0")),
    verbose_feature_names_out=False,
)
t.fit(pd.DataFrame({"x0": [None, "None"]}))
print(t.get_feature_names_out(["x0"]))
```

There's a subtle bug in the ColumnTransformer as well. If `verbose_feature_names_out` is True, it will not check names and will emit non-unique names without raising.

### Expected Results

I'd expect different feature names being generated.

### Actual Results

First code snippet
```py
['x0_None' 'x0_None']
```

Second code snippet
```py
Traceback (most recent call last):
  File "src/ohe_bug.py", line 10, in <module>
    print(t.get_feature_names_out(["x0"]))
  File "/home/pilotuser/.local/lib/python3.8/site-packages/sklearn/compose/_column_transformer.py", line 518, in get_feature_names_out
    raise ValueError(
ValueError: Output feature names: ['x0_None'] are not unique. Please set verbose_feature_names_out=True to add prefixes to feature names
```

### Versions

```shell
System:
    python: 3.8.12 (default, Sep 28 2021, 19:23:30)  [GCC 8.3.0]
executable: /usr/local/bin/python
   machine: Linux-4.14.238-182.422.amzn2.x86_64-x86_64-with-glibc2.2.5

Python dependencies:
          pip: 21.2.4
   setuptools: 46.0.0
      sklearn: 1.0.2
        numpy: 1.18.5
        scipy: 1.5.4
       Cython: None
       pandas: 1.1.1
   matplotlib: 3.2.2
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```
```

COMMENT:
I think the suggested solution for a callable in `ColumnTransformer` would be the most flexible. It would also help resolve other issues such as https://github.com/scikit-learn/scikit-learn/issues/16593

Labels: Bug, module:compose
OneHotEncoder creates non unique column names, when values None (NoneType) and "None" (str) are both present

### Describe the bug

OneHotEncoder creates non unique column names when values None (NoneType) and "None" (str) are both present in the same column.

Such policy creates downstream problems (e.g. ColumnTransformer raising because of non unique column names).

It's not 100% clear to me what's the expectation of a transformer.
Should every transformer emit unique names for features?
Why does the ColumnTransformer raise if there are non-unique names?

### Steps/Code to Reproduce

Just one hot encoder.
```py
from sklearn.preprocessing import OneHotEncoder

X = [['None'], [None]]
t = OneHotEncoder().fit(X)
feature_names = t.get_feature_names_out()
# ['x0_None' 'x0_None']

assert len(feature_names) == len(set(feature_names))
```


With ColumnTransformer
```py
import pandas as pd
from sklearn.compose import make_column_selector, make_column_transformer
from sklearn.preprocessing import OneHotEncoder

t = make_column_transformer(
    (OneHotEncoder(), make_column_selector("x0")),
    verbose_feature_names_out=False,
)
t.fit(pd.DataFrame({"x0": [None, "None"]}))
print(t.get_feature_names_out(["x0"]))
```

There's a subtle bug in the ColumnTransformer as well. If `verbose_feature_names_out` is True, it will not check names and will emit non-unique names without raising.

### Expected Results

I'd expect different feature names being generated.

### Actual Results

First code snippet
```py
['x0_None' 'x0_None']
```

Second code snippet
```py
Traceback (most recent call last):
  File "src/ohe_bug.py", line 10, in <module>
    print(t.get_feature_names_out(["x0"]))
  File "/home/pilotuser/.local/lib/python3.8/site-packages/sklearn/compose/_column_transformer.py", line 518, in get_feature_names_out
    raise ValueError(
ValueError: Output feature names: ['x0_None'] are not unique. Please set verbose_feature_names_out=True to add prefixes to feature names
```

### Versions

```shell
System:
    python: 3.8.12 (default, Sep 28 2021, 19:23:30)  [GCC 8.3.0]
executable: /usr/local/bin/python
   machine: Linux-4.14.238-182.422.amzn2.x86_64-x86_64-with-glibc2.2.5

Python dependencies:
          pip: 21.2.4
   setuptools: 46.0.0
      sklearn: 1.0.2
        numpy: 1.18.5
        scipy: 1.5.4
       Cython: None
       pandas: 1.1.1
   matplotlib: 3.2.2
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```
```

COMMENT:
@thomasjpfan I'll prepare the PR today or tomorrow so we have a starting implementation to discuss.

Labels: Bug, module:compose
OneHotEncoder creates non unique column names, when values None (NoneType) and "None" (str) are both present

### Describe the bug

OneHotEncoder creates non unique column names when values None (NoneType) and "None" (str) are both present in the same column.

Such policy creates downstream problems (e.g. ColumnTransformer raising because of non unique column names).

It's not 100% clear to me what's the expectation of a transformer.
Should every transformer emit unique names for features?
Why does the ColumnTransformer raise if there are non-unique names?

### Steps/Code to Reproduce

Just one hot encoder.
```py
from sklearn.preprocessing import OneHotEncoder

X = [['None'], [None]]
t = OneHotEncoder().fit(X)
feature_names = t.get_feature_names_out()
# ['x0_None' 'x0_None']

assert len(feature_names) == len(set(feature_names))
```


With ColumnTransformer
```py
import pandas as pd
from sklearn.compose import make_column_selector, make_column_transformer
from sklearn.preprocessing import OneHotEncoder

t = make_column_transformer(
    (OneHotEncoder(), make_column_selector("x0")),
    verbose_feature_names_out=False,
)
t.fit(pd.DataFrame({"x0": [None, "None"]}))
print(t.get_feature_names_out(["x0"]))
```

There's a subtle bug in the ColumnTransformer as well. If `verbose_feature_names_out` is True, it will not check names and will emit non-unique names without raising.

### Expected Results

I'd expect different feature names being generated.

### Actual Results

First code snippet
```py
['x0_None' 'x0_None']
```

Second code snippet
```py
Traceback (most recent call last):
  File "src/ohe_bug.py", line 10, in <module>
    print(t.get_feature_names_out(["x0"]))
  File "/home/pilotuser/.local/lib/python3.8/site-packages/sklearn/compose/_column_transformer.py", line 518, in get_feature_names_out
    raise ValueError(
ValueError: Output feature names: ['x0_None'] are not unique. Please set verbose_feature_names_out=True to add prefixes to feature names
```

### Versions

```shell
System:
    python: 3.8.12 (default, Sep 28 2021, 19:23:30)  [GCC 8.3.0]
executable: /usr/local/bin/python
   machine: Linux-4.14.238-182.422.amzn2.x86_64-x86_64-with-glibc2.2.5

Python dependencies:
          pip: 21.2.4
   setuptools: 46.0.0
      sklearn: 1.0.2
        numpy: 1.18.5
        scipy: 1.5.4
       Cython: None
       pandas: 1.1.1
   matplotlib: 3.2.2
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```
```

COMMENT:
I've created a PR linked above (https://github.com/scikit-learn/scikit-learn/pull/22506) that provides one way to fix it.

Labels: Bug, module:compose
Deprecate normalize parameter in `calibration_curve`

### Describe the workflow you want to enable

Similar to the behavior of `calibration_curve`, I would like to be able to set `CalibrationDisplay.from_predictions(normalize=True)`.

### Describe your proposed solution

Add a keyword argument `normalize` to `CalibrationDisplay.from_predictions` and pass it to `calibration_curve`. 

### Describe alternatives you've considered, if relevant

I could manually normalize the values, but given that the functionality is already present in the underlying `calibration_curve` I would say this is a relatively easy thing to adjust on the scikit-learn side.

### Additional context

_No response_

COMMENT:
Yep, it seems to be a legit request since this is just forwarding a keyword to the underlying call to `calibration_curve`.
@lucyleeow @ogrisel When creating this display, was there any discussion to omit this parameter on purpose?

Labels: Moderate, module:calibration
Deprecate normalize parameter in `calibration_curve`

### Describe the workflow you want to enable

Similar to the behavior of `calibration_curve`, I would like to be able to set `CalibrationDisplay.from_predictions(normalize=True)`.

### Describe your proposed solution

Add a keyword argument `normalize` to `CalibrationDisplay.from_predictions` and pass it to `calibration_curve`. 

### Describe alternatives you've considered, if relevant

I could manually normalize the values, but given that the functionality is already present in the underlying `calibration_curve` I would say this is a relatively easy thing to adjust on the scikit-learn side.

### Additional context

_No response_

COMMENT:
@glemaitre we had some comments about this: https://github.com/scikit-learn/scikit-learn/pull/17443/files#r670597315 

Essentially, we thought that `from_estimator` should only accept estimators with a `predict_proba` method, and we are sure `predict_proba` will output a true probability. We had also decided that we did not want to (/not a good idea to) pass output of `decision_function` to `calibration_curve`, which probably boils down to @NicolasHug's comment here: https://github.com/scikit-learn/scikit-learn/pull/17443#discussion_r440103386, though that whole thread is probably relevant.

Labels: Moderate, module:calibration
Deprecate normalize parameter in `calibration_curve`

### Describe the workflow you want to enable

Similar to the behavior of `calibration_curve`, I would like to be able to set `CalibrationDisplay.from_predictions(normalize=True)`.

### Describe your proposed solution

Add a keyword argument `normalize` to `CalibrationDisplay.from_predictions` and pass it to `calibration_curve`. 

### Describe alternatives you've considered, if relevant

I could manually normalize the values, but given that the functionality is already present in the underlying `calibration_curve` I would say this is a relatively easy thing to adjust on the scikit-learn side.

### Additional context

_No response_

COMMENT:
If I understand correctly from the thread, the intuition behind the decision is that one should not rely on simple scaling because it is not proper calibration? I think that makes sense. Though it is a bit annoying for lazy me,  even with manual scaling, `CalibrationDisplay` saves me more lines of code compared to `calibration_curve`.

I'm not sure if it matters, but I was specifically referring to `from_predictions` rather than `from_estimator`. I try to asses the extent to which a set of given risk scores (that go from 1 to 9) are *similarly* calibrated for different subgroups, which can be relevant for fairness assessments. I am not interested in actually deriving a calibrated classifier.

By the way, does this mean that `normalize` in `calibration_curve` will at some point also be deprecated or will it be kept for backward compatibility?

Labels: Moderate, module:calibration
Deprecate normalize parameter in `calibration_curve`

### Describe the workflow you want to enable

Similar to the behavior of `calibration_curve`, I would like to be able to set `CalibrationDisplay.from_predictions(normalize=True)`.

### Describe your proposed solution

Add a keyword argument `normalize` to `CalibrationDisplay.from_predictions` and pass it to `calibration_curve`. 

### Describe alternatives you've considered, if relevant

I could manually normalize the values, but given that the functionality is already present in the underlying `calibration_curve` I would say this is a relatively easy thing to adjust on the scikit-learn side.

### Additional context

_No response_

COMMENT:
OK, I went into the discussion in the PR and discussed it as well with @ogrisel IRL.

Methodologically, a calibration curve should only be used with a probability. Using `calibration_curve` with `normalize=True` is equivalent to a naive linear calibration with additional clipping for value above/under max/min.

So I think that we don't want people using this type of naive calibration without doing it explicitly.

> I try to asses the extent to which a set of given risk scores (that go from 1 to 9) are similarly calibrated for different subgroups, which can be relevant for fairness assessments. I am not interested in actually deriving a calibrated classifier.

I think that in your case it boils down to how you want to relate the risk scores to a probability. If you want a linear mapping then `normalize=True` was the right move. However, you could as well use a sigmoid transform centred on score 5 to express a probability. Since I don't have the background on what are the scores, this is difficult to say what is the right choice to transform the score into probability but it shows that using the naive normalization implicitly could be dangerous in general.

> By the way, does this mean that normalize in calibration_curve will at some point also be deprecated or will it be kept for backward compatibility?

I would be in favour of deprecating `normalize` as well with an explicit warning that encourages users to provide probabilities that could be obtained through a calibrated classifier if the output is a decision function.

Labels: Moderate, module:calibration
Deprecate normalize parameter in `calibration_curve`

### Describe the workflow you want to enable

Similar to the behavior of `calibration_curve`, I would like to be able to set `CalibrationDisplay.from_predictions(normalize=True)`.

### Describe your proposed solution

Add a keyword argument `normalize` to `CalibrationDisplay.from_predictions` and pass it to `calibration_curve`. 

### Describe alternatives you've considered, if relevant

I could manually normalize the values, but given that the functionality is already present in the underlying `calibration_curve` I would say this is a relatively easy thing to adjust on the scikit-learn side.

### Additional context

_No response_

COMMENT:
> Methodologically, a calibration curve should only be used with a probability. Using calibration_curve with normalize=True is equivalent to a naive linear calibration with additional clipping for value above/under max/min.

> So I think that we don't want people using this type of naive calibration without doing it explicitly.

I agree. Happy to deprecate as well.

Labels: Moderate, module:calibration
Deprecate normalize parameter in `calibration_curve`

### Describe the workflow you want to enable

Similar to the behavior of `calibration_curve`, I would like to be able to set `CalibrationDisplay.from_predictions(normalize=True)`.

### Describe your proposed solution

Add a keyword argument `normalize` to `CalibrationDisplay.from_predictions` and pass it to `calibration_curve`. 

### Describe alternatives you've considered, if relevant

I could manually normalize the values, but given that the functionality is already present in the underlying `calibration_curve` I would say this is a relatively easy thing to adjust on the scikit-learn side.

### Additional context

_No response_

COMMENT:
+1 for deprecation of `normalize` in `calibration_curve` as well. BTW, this is a suprising twist on this stage: the request turned into a deprecation decision 😄

Labels: Moderate, module:calibration
Deprecate normalize parameter in `calibration_curve`

### Describe the workflow you want to enable

Similar to the behavior of `calibration_curve`, I would like to be able to set `CalibrationDisplay.from_predictions(normalize=True)`.

### Describe your proposed solution

Add a keyword argument `normalize` to `CalibrationDisplay.from_predictions` and pass it to `calibration_curve`. 

### Describe alternatives you've considered, if relevant

I could manually normalize the values, but given that the functionality is already present in the underlying `calibration_curve` I would say this is a relatively easy thing to adjust on the scikit-learn side.

### Additional context

_No response_

COMMENT:
> +1 for deprecation of normalize in calibration_curve as well. BTW, this is a suprising twist on this stage: the request turned into a deprecation decision

lol, happy to help :smile:

Labels: Moderate, module:calibration
Deprecate normalize parameter in `calibration_curve`

### Describe the workflow you want to enable

Similar to the behavior of `calibration_curve`, I would like to be able to set `CalibrationDisplay.from_predictions(normalize=True)`.

### Describe your proposed solution

Add a keyword argument `normalize` to `CalibrationDisplay.from_predictions` and pass it to `calibration_curve`. 

### Describe alternatives you've considered, if relevant

I could manually normalize the values, but given that the functionality is already present in the underlying `calibration_curve` I would say this is a relatively easy thing to adjust on the scikit-learn side.

### Additional context

_No response_

COMMENT:
I will work on this.

Labels: Moderate, module:calibration
DummyRegressor converts some params to NumPy after fit()

### Describe the bug

The parameters of the DummyRegressor get converted into NumPy-format after calling .fit(). This is disadvantageous if the parameters are to be extracted and written in a JSON format, e.g., to save configurations.

### Steps/Code to Reproduce

```
from sklearn.dummy import DummyRegressor

dummy = DummyRegressor(constant=0.0, strategy="constant")
print(dummy.get_params())

dummy.fit(X=[[1,1], [2,2], [3,3]], y=[[0], [0], [0]])
print(dummy.get_params())
```

### Expected Results

After fitting the DummyRegressor, params are not converted into NumPy format.

before fit
```
{'constant': 0.0, 'quantile': None, 'strategy': 'constant'}
```

after fit
```
{'constant': 0.0, 'quantile': None, 'strategy': 'constant'}
```

### Actual Results

before fit
```
{'constant': 0.0, 'quantile': None, 'strategy': 'constant'}
```

after fit
```
{'constant': array(0.), 'quantile': None, 'strategy': 'constant'}
```

### Versions

```shell
System:
    python: 3.8.5 (default, Sep  4 2020, 07:30:14)  [GCC 7.3.0]
executable: ***
   machine: Linux-5.4.0-96-generic-x86_64-with-glibc2.10

Python dependencies:
          pip: 20.1.1
   setuptools: 47.1.0
      sklearn: 1.0.2
        numpy: 1.19.3
        scipy: 1.6.0
       Cython: 0.29.25
       pandas: 1.2.1
   matplotlib: 3.3.4
       joblib: 1.0.0
threadpoolctl: 2.1.0

Built with OpenMP: True
```
```

COMMENT:
Indeed, this is against our API as well. If required at predict, we should store the validated value in a private attribute.

Labels: Bug
Clarification around `cross_val_predict` cross validator.

### Describe the issue linked to the documentation

This is not necessarily true for [cross_val_predict](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_predict.html), right?: 
_Each sample belongs to exactly one test set..._

For example, what if your cv was a version of `ShuffleSplit`?


### Suggest a potential alternative/fix

I think either this should be omitted entirely or refer to the fact that the `cv` kwarg dicates what's in which test set.

COMMENT:
Nevermind ~~ documentation still potentially unclear, but here is a new suggestion discovered only when reading the User Guide.... refer to this to confirm that cv cannot be ShuffleSplit: https://scikit-learn.org/stable/modules/cross_validation.html#obtaining-predictions-by-cross-validation

Labels: Documentation, Needs Triage
Clarification around `cross_val_predict` cross validator.

### Describe the issue linked to the documentation

This is not necessarily true for [cross_val_predict](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_predict.html), right?: 
_Each sample belongs to exactly one test set..._

For example, what if your cv was a version of `ShuffleSplit`?


### Suggest a potential alternative/fix

I think either this should be omitted entirely or refer to the fact that the `cv` kwarg dicates what's in which test set.

COMMENT:
Yep you cannot pass a `SuffleSplit`. It will raise an error.

Labels: Documentation, Needs Triage