Talip7/scikit-learn-issues-multilabel huggingface dataset:
sklearn.preprocessing.StandardScaler() needs a degrees of freedom tunable parameter for scale_

### Describe the workflow you want to enable

I'm scaling a dataset using the standard procedure.

```
from sklearn.preprocessing import StandardScaler

sc = StandardScaler()
sc.fit(data)
sc.transform(data)
```

I'm comparing the results I get with the results from the R function `scale()`. They are subtly different. Tracking down the difference, it turns out sklearn uses a different definition for the standard deviation, one that differs from R in the degrees of freedom assumed. sklearn assumes ddof=0, while R assumes ddof=1.

https://stackoverflow.com/questions/27296387/difference-between-r-scale-and-sklearn-preprocessing-scale/72758497

I do not wish to debate which approach is "more correct". I am fine with sklearn continuing with the current approach as the default. I only want to suggest that, for those who think the slight DoF correction is necessary, `StandardScaler()` should provide an option to tune the degrees of freedom, either in the initial `StandardScaler()` call, or later in the `fit()` call, or wherever it is deemed more appropriate.

### Describe your proposed solution

Basically, leave the default as-is. Introduce a new parameter, let's call it `ddof`, with a default value of 0, but allow the user to specify a different value for it if they need it.

Or have a boolean switch, by default False, which when True enables the Bessel correction, which is the same as using N-1 instead of N for the standard deviation.

I do not know enough about the sklearn internals to judge which solution is better.

### Describe alternatives you've considered, if relevant

This workaround forces `StandardScaler()` to use ddof=1, but it's an extra step and it computes the STD twice. With large datasets, that would slow down the computation and it's wasteful.

```
from sklearn.preprocessing import StandardScaler
import numpy as np

sc = StandardScaler()
sc.fit(data)
sc.scale_ = np.std(data, axis=0, ddof=1).to_list()
sc.transform(data)
```

### Additional context

https://en.wikipedia.org/wiki/Bessel%27s_correction

COMMENT:
@moralech-mountunion Please provide a ML use case where this difference actually matters.
As scikit-learn maintainers with experience from real life applications, we are convinced that this is hard to find.
Note that you can wrap `np.std` with the desired dof in a `FunctionTransformer`.

Labels: New Feature, module:preprocessing, Needs Decision - Include Feature
sklearn.preprocessing.StandardScaler() needs a degrees of freedom tunable parameter for scale_

### Describe the workflow you want to enable

I'm scaling a dataset using the standard procedure.

```
from sklearn.preprocessing import StandardScaler

sc = StandardScaler()
sc.fit(data)
sc.transform(data)
```

I'm comparing the results I get with the results from the R function `scale()`. They are subtly different. Tracking down the difference, it turns out sklearn uses a different definition for the standard deviation, one that differs from R in the degrees of freedom assumed. sklearn assumes ddof=0, while R assumes ddof=1.

https://stackoverflow.com/questions/27296387/difference-between-r-scale-and-sklearn-preprocessing-scale/72758497

I do not wish to debate which approach is "more correct". I am fine with sklearn continuing with the current approach as the default. I only want to suggest that, for those who think the slight DoF correction is necessary, `StandardScaler()` should provide an option to tune the degrees of freedom, either in the initial `StandardScaler()` call, or later in the `fit()` call, or wherever it is deemed more appropriate.

### Describe your proposed solution

Basically, leave the default as-is. Introduce a new parameter, let's call it `ddof`, with a default value of 0, but allow the user to specify a different value for it if they need it.

Or have a boolean switch, by default False, which when True enables the Bessel correction, which is the same as using N-1 instead of N for the standard deviation.

I do not know enough about the sklearn internals to judge which solution is better.

### Describe alternatives you've considered, if relevant

This workaround forces `StandardScaler()` to use ddof=1, but it's an extra step and it computes the STD twice. With large datasets, that would slow down the computation and it's wasteful.

```
from sklearn.preprocessing import StandardScaler
import numpy as np

sc = StandardScaler()
sc.fit(data)
sc.scale_ = np.std(data, axis=0, ddof=1).to_list()
sc.transform(data)
```

### Additional context

https://en.wikipedia.org/wiki/Bessel%27s_correction

COMMENT:
@moralech-mountunion This will probably be my last comment on this discussion. My impression is that you have an incomplete understanding of machine learning. Scaling is usually part of a preprocessing pipeline. For tree based models (supervised ML), for instance, scaling has basically 0 (really almost exactly 0) influence on the predictions of the tree(s). Same story for unpenalized linear models. Then the actual ML algorithms in scikit-learn usually have a high numerical stability and we have a huge test suite to ensure correctness to as many bits as maintainability and runtime permits.

Labels: New Feature, module:preprocessing, Needs Decision - Include Feature
⚠️ CI failed on macOS.pylatest_conda_forge_mkl ⚠️

**CI failed on [macOS.pylatest_conda_forge_mkl](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=43822&view=logs&j=97641769-79fb-5590-9088-a30ce9b850b9)**
- test_fastica_eigh_low_rank_warning[69]

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=43833&view=logs&j=97641769-79fb-5590-9088-a30ce9b850b9)

Labels: Bug
⚠️ CI failed on macOS.pylatest_conda_forge_mkl ⚠️

**CI failed on [macOS.pylatest_conda_forge_mkl](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=43822&view=logs&j=97641769-79fb-5590-9088-a30ce9b850b9)**
- test_fastica_eigh_low_rank_warning[69]

COMMENT:
The test initially failed in `global_random_seed=69` and passed again for another random seed.

Labels: Bug
⚠️ CI failed on macOS.pylatest_conda_forge_mkl ⚠️

**CI failed on [macOS.pylatest_conda_forge_mkl](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=43822&view=logs&j=97641769-79fb-5590-9088-a30ce9b850b9)**
- test_fastica_eigh_low_rank_warning[69]

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=43840&view=logs&j=97641769-79fb-5590-9088-a30ce9b850b9)

Labels: Bug
Erreneous Avg. Loss calculation

https://github.com/scikit-learn/scikit-learn/blob/80598905e517759b4696c74ecc35c6e2eb508cff/sklearn/linear_model/_sgd_fast.pyx#L633-L636

The avg. loss should be `sumloss/count` instead of `sumloss/n_samples`. This is because the validation data points are ignored during loss calculation and hence are not added in `sumloss`.

COMMENT:
Is this bug only in this print method or somewhere it the actual algoritm/optimization routine?

Labels: Bug, module:linear_model
Erreneous Avg. Loss calculation

https://github.com/scikit-learn/scikit-learn/blob/80598905e517759b4696c74ecc35c6e2eb508cff/sklearn/linear_model/_sgd_fast.pyx#L633-L636

The avg. loss should be `sumloss/count` instead of `sumloss/n_samples`. This is because the validation data points are ignored during loss calculation and hence are not added in `sumloss`.

COMMENT:
The variable `sumloss` is not directly used in the algorithm/optimization routine. It is primarily used for verbosity and the logic that drives early stopping. For example in the snippet below, the correct condition to check should be `sumloss > best_loss - tol * count` instead of `n_samples`. The current logic has the effect of overscaling the tolerance parameter provided by user which is completely unintended. 

https://github.com/scikit-learn/scikit-learn/blob/80598905e517759b4696c74ecc35c6e2eb508cff/sklearn/linear_model/_sgd_fast.pyx#L658-L659

Labels: Bug, module:linear_model
Erreneous Avg. Loss calculation

https://github.com/scikit-learn/scikit-learn/blob/80598905e517759b4696c74ecc35c6e2eb508cff/sklearn/linear_model/_sgd_fast.pyx#L633-L636

The avg. loss should be `sumloss/count` instead of `sumloss/n_samples`. This is because the validation data points are ignored during loss calculation and hence are not added in `sumloss`.

COMMENT:
@Harsh14901 Thanks for reporting. Do you want to give it a try to fix it in a PR?

Labels: Bug, module:linear_model
Make `penalty` parameter consistent for `None`

We are inconsistent with the `penalty` parameter for the linear models.
For the SGD and Perceptron, we will accept `penalty=None` while this is not the case in `LogisticRegression`. In `LogisticRegression`, we must provide the string `'none'`.

I assume we could make everything consistent by accepting both `None` and `'none'`.

@jeremiedbb WDYT?

COMMENT:
The test can be easily handled by the common validation parameter if we decorate properly the estimator.

Labels: API
Make `penalty` parameter consistent for `None`

We are inconsistent with the `penalty` parameter for the linear models.
For the SGD and Perceptron, we will accept `penalty=None` while this is not the case in `LogisticRegression`. In `LogisticRegression`, we must provide the string `'none'`.

I assume we could make everything consistent by accepting both `None` and `'none'`.

@jeremiedbb WDYT?

COMMENT:
> I assume we could make everything consistent by accepting both None and 'none'

I'd just accept `None` but it would require a deprecation cycle so no strong opinion

Labels: API
Make `penalty` parameter consistent for `None`

We are inconsistent with the `penalty` parameter for the linear models.
For the SGD and Perceptron, we will accept `penalty=None` while this is not the case in `LogisticRegression`. In `LogisticRegression`, we must provide the string `'none'`.

I assume we could make everything consistent by accepting both `None` and `'none'`.

@jeremiedbb WDYT?

COMMENT:
So it means that we would deprecate and ask to use `None` in `LogisticRegression`.

Labels: API
Make `penalty` parameter consistent for `None`

We are inconsistent with the `penalty` parameter for the linear models.
For the SGD and Perceptron, we will accept `penalty=None` while this is not the case in `LogisticRegression`. In `LogisticRegression`, we must provide the string `'none'`.

I assume we could make everything consistent by accepting both `None` and `'none'`.

@jeremiedbb WDYT?

COMMENT:
I'm okay with the deprecation here and go with `None`.

Labels: API
Add set difference to `validate_parameter_constraints` `ValueError`

### Describe the workflow you want to enable

Currently the `ValueError` protuced by `validate_parameter_constraints` can get very long and unruly when there are a lot of validation parameters in either the passed constraints, or in the params to be validated. This can make debugging a bit strained. I think it would be a huge help to also include the set differences to more clearly explain what may be happening.

### Describe your proposed solution

Something along the lines of:

```python
...
if params.keys() != parameter_constraints.keys():
    raise ValueError(
        f"The parameter constraints {list(parameter_constraints.keys())} do not "
        f"match the parameters to validate {list(params.keys())}.\n\n"
        f"Consider the extra constraints {set(parameter_constraints.keys())-set(params.keys())}"
        f" and missing expected constraints {set(params.keys())-set(parameter_constraints.keys())}"
    )
```

### Describe alternatives you've considered, if relevant

N/A

### Additional context

```python-traceback


ValueError: The parameter constraints ['fit_intercept', 'max_iter', 'tol', 'shuffle', 'verbose', 'random_state', 'warm_start', 'average', 'loss', 'n_jobs', 'class_weight', 'early_stopping', 'validation_fraction', 'n_iter_no_change'] do not match the parameters to validate ['alpha', 'class_weight', 'early_stopping', 'eta0', 'fit_intercept', 'l1_ratio', 'max_iter', 'n_iter_no_change', 'n_jobs', 'penalty', 'random_state', 'shuffle', 'tol', 'validation_fraction', 'verbose', 'warm_start'].
E
E           Consider the extra constraints {'average', 'loss'} and missing expected constraints {'l1_ratio', 'penalty', 'alpha', 'eta0'}
```

I believe the set differences make this **_much_** more readable, and should be a very small/simple inclusion.

COMMENT:
@jeremiedbb Wondering your opinion on this

Labels: New Feature
Add set difference to `validate_parameter_constraints` `ValueError`

### Describe the workflow you want to enable

Currently the `ValueError` protuced by `validate_parameter_constraints` can get very long and unruly when there are a lot of validation parameters in either the passed constraints, or in the params to be validated. This can make debugging a bit strained. I think it would be a huge help to also include the set differences to more clearly explain what may be happening.

### Describe your proposed solution

Something along the lines of:

```python
...
if params.keys() != parameter_constraints.keys():
    raise ValueError(
        f"The parameter constraints {list(parameter_constraints.keys())} do not "
        f"match the parameters to validate {list(params.keys())}.\n\n"
        f"Consider the extra constraints {set(parameter_constraints.keys())-set(params.keys())}"
        f" and missing expected constraints {set(params.keys())-set(parameter_constraints.keys())}"
    )
```

### Describe alternatives you've considered, if relevant

N/A

### Additional context

```python-traceback


ValueError: The parameter constraints ['fit_intercept', 'max_iter', 'tol', 'shuffle', 'verbose', 'random_state', 'warm_start', 'average', 'loss', 'n_jobs', 'class_weight', 'early_stopping', 'validation_fraction', 'n_iter_no_change'] do not match the parameters to validate ['alpha', 'class_weight', 'early_stopping', 'eta0', 'fit_intercept', 'l1_ratio', 'max_iter', 'n_iter_no_change', 'n_jobs', 'penalty', 'random_state', 'shuffle', 'tol', 'validation_fraction', 'verbose', 'warm_start'].
E
E           Consider the extra constraints {'average', 'loss'} and missing expected constraints {'l1_ratio', 'penalty', 'alpha', 'eta0'}
```

I believe the set differences make this **_much_** more readable, and should be a very small/simple inclusion.

COMMENT:
I felt the same difficulty while reviewing some PRs. I think this would be a good addition the common tests `check_param_validation` and `def test_function_param_validation`

Labels: New Feature
SimpleImputer does not implement `get_feature_names_out`

### Describe the workflow you want to enable

I noticed that [SimpleImputer](https://github.com/scikit-learn/scikit-learn/blob/159cb46c5672b0ba87d0ba80c6b80ec1aa5fda32/sklearn/impute/_base.py#L132) is still not supporting `get_feature_names_out`.  In looking at issues like https://github.com/scikit-learn/scikit-learn/issues/21308 and https://github.com/scikit-learn/scikit-learn/issues/21688, I got the impression this may have been an oversight -- particularly since it looks like it's easy to fix this with `_OneToOneFeatureMixin`.

Is this just an omission or is there more to why that was not yet included?  I couldn't find any other issues speaking to the latter.

### Describe your proposed solution

Add `_OneToOneFeatureMixin` to `SimpleImputer`.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
`SimpleImputer` defines `get_feature_names_out` here: (It requires a custom implementation because it is not always one to one)

https://github.com/scikit-learn/scikit-learn/blob/d7c38282839d09676c49ac60fdd67af89d61e79c/sklearn/impute/_base.py#L645-L646

With scikit-learn 1.1, the following works:

```python
import numpy as np
from sklearn.impute import SimpleImputer
import pandas as pd

X = pd.DataFrame([[7, 2, 3], [4, np.nan, 6], [10, 5, 9]], columns=["feat1", "feat2", "feat3"])
imp_mean = SimpleImputer(missing_values=np.nan, strategy='mean')
imp_mean.fit(X)

print(imp_mean.get_feature_names_out())
# ['feat1' 'feat2' 'feat3']

```

Labels: New Feature, Needs Triage
Add `n_jobs` and `random_state` to global config

### Describe the workflow you want to enable

I would like to be able to set `n_jobs=-1` in one place and have this take effect in any function with an `n_jobs` parameter. Same for `random_state`.

Perhaps there are other parameters that fit the theme: "if a user sets this in one instance, they _probably_ want to set it in all instances".

### Describe your proposed solution

Expand the accepted parameters of [sklearn.set_config](https://scikit-learn.org/stable/modules/generated/sklearn.set_config.html#sklearn.set_config), update all functions to fall back to the config value if the parameter isn't passed.

This would require changing a default of `None` to a sentinel (ala `_NoValue` in NumPy), to allow a user to override the global config with the value `None`, while still allowing the code to detect if the argument was passed.

Rudimentary mockup:
```py
_sentinel = object()

config = {}


def resolve_arg_value(arg_name, passed_value, default_value):
    if passed_value is not _sentinel:
        return passed_value
    
    if arg_name in config:
        return config[arg_name]
    
    return default_value


def do_something(random_state=_sentinel):
    random_state = resolve_arg_value("random_state", random_state, None)
    print(f"{random_state!s}")


do_something('Hi')  # 'Hi'
do_something()  # None
do_something(None)  # None

config['random_state'] = 77

do_something('Hi')  # 'Hi'
do_something()  # 77
do_something(None)  # None

config.pop('random_state')

do_something('Hi')  # 'Hi'
do_something()  # None
do_something(None)  # None
```

I'll admit, having to add something like `resolve_arg_value("random_state", random_state, None)` to tons of functions sounds painful, but I think for the user, being able to set and forget a random state/other params, potentially based on an environment variable would be nice.
```py
sklearn.set_config(
    random_state=None if os.environ["PROD"] else 0,
    n_jobs=-1,
)
```


### Describe alternatives you've considered, if relevant

_No response_

### Additional context

I see that the tests get a global random seed. https://github.com/scikit-learn/scikit-learn/pull/22749

COMMENT:
For `n_jobs`, we ruled out the possibility here: https://github.com/scikit-learn/scikit-learn/issues/23253. It is already possible to do so with the `parallel_backend` of `joblib`. However, be extremely careful with `n_jobs=-1`.

For the `random_state`, isn't it enough to seed the singleton random generator from NumPy using [`np.random.seed`](https://numpy.org/doc/stable/reference/random/generated/numpy.random.RandomState.seed.html#numpy.random.RandomState.seed) and let all `random_state=None`. In this case, the results will be reproducible since `random_state=None` will use the global singleton.

Labels: New Feature, API, Needs Decision
Add `n_jobs` and `random_state` to global config

### Describe the workflow you want to enable

I would like to be able to set `n_jobs=-1` in one place and have this take effect in any function with an `n_jobs` parameter. Same for `random_state`.

Perhaps there are other parameters that fit the theme: "if a user sets this in one instance, they _probably_ want to set it in all instances".

### Describe your proposed solution

Expand the accepted parameters of [sklearn.set_config](https://scikit-learn.org/stable/modules/generated/sklearn.set_config.html#sklearn.set_config), update all functions to fall back to the config value if the parameter isn't passed.

This would require changing a default of `None` to a sentinel (ala `_NoValue` in NumPy), to allow a user to override the global config with the value `None`, while still allowing the code to detect if the argument was passed.

Rudimentary mockup:
```py
_sentinel = object()

config = {}


def resolve_arg_value(arg_name, passed_value, default_value):
    if passed_value is not _sentinel:
        return passed_value
    
    if arg_name in config:
        return config[arg_name]
    
    return default_value


def do_something(random_state=_sentinel):
    random_state = resolve_arg_value("random_state", random_state, None)
    print(f"{random_state!s}")


do_something('Hi')  # 'Hi'
do_something()  # None
do_something(None)  # None

config['random_state'] = 77

do_something('Hi')  # 'Hi'
do_something()  # 77
do_something(None)  # None

config.pop('random_state')

do_something('Hi')  # 'Hi'
do_something()  # None
do_something(None)  # None
```

I'll admit, having to add something like `resolve_arg_value("random_state", random_state, None)` to tons of functions sounds painful, but I think for the user, being able to set and forget a random state/other params, potentially based on an environment variable would be nice.
```py
sklearn.set_config(
    random_state=None if os.environ["PROD"] else 0,
    n_jobs=-1,
)
```


### Describe alternatives you've considered, if relevant

_No response_

### Additional context

I see that the tests get a global random seed. https://github.com/scikit-learn/scikit-learn/pull/22749

COMMENT:
Thanks, `joblib` suits my needs!

Re `random_state`, `np.random.seed` will work, thanks for that. Although [the docs specifically warn against it](https://scikit-learn.org/stable/common_pitfalls.html#getting-reproducible-results-across-multiple-executions) and recommend passing the `random_state` arg to every function that takes it.

So the goal of my global config suggestion was to allow the 'recommended' way of getting reproducible results without the clutter of passing a `random_state` parameter to all the scikit-learn functions that take it (and the mental overhead of thinking about which functions take it and which don't).

Also if scikit-learn one day moves from the legacy NumPy `RandomState` to the new `Generator`, a global config would provide an upgrade path to all the users who were using the discouraged `np.random.seed` technique.

Labels: New Feature, API, Needs Decision