Talip7/scikit-learn-issues-multilabel huggingface dataset:
SGDClassifier should use sparse representation for coefficients

### Describe the workflow you want to enable

`SGDClassifier` should take a `sparse_coef` initial parameter (default false).  If set to `True`, the initial `coef` will be sparse.  For models like `HashVectorizer` (default 1M features), having a dense coefficient is very inefficient.

### Describe your proposed solution

See above.

### Describe alternatives you've considered, if relevant

Automatically infer `coef_`'s sparsity based on input's spasity!

### Additional context

_No response_

COMMENT:
Storing the coefficients in a sparse matrix would require a new implementation of `sklearn.utils._weight_vector.WeightVector`. This is not trivial, and would require quite a bit of experimentation to make it efficient and to make sure it is actually useful. In particular, it is not clear to me if the coefficients would stay sparse during the optimization, which would defeat the purpose of the sparse format.

Because each SGD update is parallel to the drawn sample, if the drawn sample is sparse then the update of the coefficients is sparse as well. However, because different samples are sparse over different features, the coefficients would not stay sparse very long. A strong L1 penalty induces sparsity in the coefficients at optimum, but I am not sure that a L1 penalty actually induces sparsity during the entire optimization process. Actually, because the SGD solver is very slow for converging to the exact optimum, I wonder if the coefficients are really sparse even at the end of the algorithm.

Labels: New Feature, Needs Triage
SGDClassifier should use sparse representation for coefficients

### Describe the workflow you want to enable

`SGDClassifier` should take a `sparse_coef` initial parameter (default false).  If set to `True`, the initial `coef` will be sparse.  For models like `HashVectorizer` (default 1M features), having a dense coefficient is very inefficient.

### Describe your proposed solution

See above.

### Describe alternatives you've considered, if relevant

Automatically infer `coef_`'s sparsity based on input's spasity!

### Additional context

_No response_

COMMENT:
Seems like we're not going to implement this then / it isn't clear what the benefits are.

Labels: New Feature, Needs Triage
Custom Tie Breaking Criterion Voting Ensemble

### Describe the bug

We were wondering whether there is anything regarding customise the tie breaking criterion the Voting ensemble does ?

### Steps/Code to Reproduce

N/A

### Expected Results

N/A

### Actual Results

N/A

### Versions

```shell
N/A
```

COMMENT:
Closed as not planned. Redirected to #26399 26399

Labels: Bug, Needs Triage
GridSearchCV support callback for MLFlow

### Describe the workflow you want to enable

I would like to save off the results of all runs in GridSearchCV to MLFlow.  MLFlow 

```python
for param in params:
    with mlflow.start_run():
        est = ElasticNet(**param)
        est.fit(train_x, train_y)
        metrics = est.score(test_x, test_y)
        mlflow.log_params(param)
        mlflow.log_metrics(metrics)
        mlflow.sklearn.log_model(est, "model")
```

See https://mlflow.org/docs/latest/tutorials-and-examples/tutorial.html for more details:

I would like to use `GridSearchCV` to do the above because it comes with many other features (e.g. `HalvingGridSearchCV`, multi-threading, etc ...)[](url)

### Describe your proposed solution

A callback parameter to `GridSearchCV`.  Perhaps
```python
def log_candidate(model, test_x, test_y):
  with mlflow.start_run():
        mlflow.log_params(model.get_params())
        mlflow.log_metrics(metrics)
        mlflow.sklearn.log_model(est, "model")
```

### Describe alternatives you've considered, if relevant

To hack the scorer for this purpose: https://danielhnyk.cz/adding-callback-to-a-sklearn-gridsearch/

This is suboptimal because:
1. If you want to return multiple metrics, you cannot save multiple scores using the provided API.  This is because we have to pass multiple **scorer**s, not a function that generates multiple scores.
2. Enabling `return_train_score` will call the scorer callback too many times and it is not easy to distinguish between the training and testing scoring.

### Additional context

_No response_

COMMENT:
Hi @tianhuil! Adding a callback API is a fairly large undertaking, and indeed already in progress (https://github.com/scikit-learn/scikit-learn/pull/22000)!

I'll leave this issue open for now, since afaik it is a new/unique use-case and is helpful to keep in mind, but bear in mind that this feature is probably not going to be released for some time and still requires much work.

Labels: New Feature
NMF fit transform without updating H should not require the user to input "n_components"

The `_fit_transform` function of the `_nmf` module has the option to set `update_H=False`, where the H matrix is left constant. the private method `_fit_transform` is called by the exposed `non_negative_factorization` function.
In a scenario I've encountered, the user provides the H matrix, meaning the number of components is known a-prior, and there is no reason for the algorithm to run the lines
```
        if self._n_components is None:
            self._n_components = X.shape[1]
``` 
and raise an error later in the `_check_w_h`


https://github.com/scikit-learn/scikit-learn/blob/f5ec34e0f76277ba6d0a77d3033db0af83899b64/sklearn/decomposition/_nmf.py#LL1188C19-L1188C19

COMMENT:
Hi @yotamcons, the ``fit_transform`` method of NMF does not expose the option ``update_H``. It's the private method ``_fit_transform`` that does expose it, but it's there for internal purpose, so it's advised not to call it directly. I f you really want to use it, you need to set n_components appropriately.

Labels: help wanted, module:decomposition
NMF fit transform without updating H should not require the user to input "n_components"

The `_fit_transform` function of the `_nmf` module has the option to set `update_H=False`, where the H matrix is left constant. the private method `_fit_transform` is called by the exposed `non_negative_factorization` function.
In a scenario I've encountered, the user provides the H matrix, meaning the number of components is known a-prior, and there is no reason for the algorithm to run the lines
```
        if self._n_components is None:
            self._n_components = X.shape[1]
``` 
and raise an error later in the `_check_w_h`


https://github.com/scikit-learn/scikit-learn/blob/f5ec34e0f76277ba6d0a77d3033db0af83899b64/sklearn/decomposition/_nmf.py#LL1188C19-L1188C19

COMMENT:
Sorry for the misleading writing, the problem is that `non_negative_factorization` internally calls `_fit_transform` and causes the said issue. I've edited the issue to make it clearer.

Labels: help wanted, module:decomposition
NMF fit transform without updating H should not require the user to input "n_components"

The `_fit_transform` function of the `_nmf` module has the option to set `update_H=False`, where the H matrix is left constant. the private method `_fit_transform` is called by the exposed `non_negative_factorization` function.
In a scenario I've encountered, the user provides the H matrix, meaning the number of components is known a-prior, and there is no reason for the algorithm to run the lines
```
        if self._n_components is None:
            self._n_components = X.shape[1]
``` 
and raise an error later in the `_check_w_h`


https://github.com/scikit-learn/scikit-learn/blob/f5ec34e0f76277ba6d0a77d3033db0af83899b64/sklearn/decomposition/_nmf.py#LL1188C19-L1188C19

COMMENT:
Thanks for the clarification. Indeed, we check that the shape are consistent. `n_components` is only determined by the `n_components` parameter so it must be set. The same behavior occurs when you set `init="custom"` and provide H and W with wrong shapes.

I wouldn't change this behavior. Instead we could probably improve the description of the W and H parameters to mention that their shapes must be in line with `n_components`. Would you like to submit a PR ?

Labels: help wanted, module:decomposition
NMF fit transform without updating H should not require the user to input "n_components"

The `_fit_transform` function of the `_nmf` module has the option to set `update_H=False`, where the H matrix is left constant. the private method `_fit_transform` is called by the exposed `non_negative_factorization` function.
In a scenario I've encountered, the user provides the H matrix, meaning the number of components is known a-prior, and there is no reason for the algorithm to run the lines
```
        if self._n_components is None:
            self._n_components = X.shape[1]
``` 
and raise an error later in the `_check_w_h`


https://github.com/scikit-learn/scikit-learn/blob/f5ec34e0f76277ba6d0a77d3033db0af83899b64/sklearn/decomposition/_nmf.py#LL1188C19-L1188C19

COMMENT:
As the given `H` already holds the information regarding the `n_components`, wouldn't it be preferred to set `n_components = H.shape[0]`?
this is especially true if the users haven't provided `n_components` themselves

Labels: help wanted, module:decomposition
NMF fit transform without updating H should not require the user to input "n_components"

The `_fit_transform` function of the `_nmf` module has the option to set `update_H=False`, where the H matrix is left constant. the private method `_fit_transform` is called by the exposed `non_negative_factorization` function.
In a scenario I've encountered, the user provides the H matrix, meaning the number of components is known a-prior, and there is no reason for the algorithm to run the lines
```
        if self._n_components is None:
            self._n_components = X.shape[1]
``` 
and raise an error later in the `_check_w_h`


https://github.com/scikit-learn/scikit-learn/blob/f5ec34e0f76277ba6d0a77d3033db0af83899b64/sklearn/decomposition/_nmf.py#LL1188C19-L1188C19

COMMENT:
Suppose you set `init="custom"` and provide W and H with shapes that don't match. Which one would you take ? I think the best solution is to raise an error in that situation. Another example: if you set `update_H=False` and set `n_components` but provide H with an non matching shape. I would also raise an error here.

`n_components=None` doesn't mean ignored, it just means `n_components=n_features`. I don't think it should generate a different behavior than setting any other value regarding matching shapes. What prevents you to set `n_components=H.shape[0]` ?

Labels: help wanted, module:decomposition
NMF fit transform without updating H should not require the user to input "n_components"

The `_fit_transform` function of the `_nmf` module has the option to set `update_H=False`, where the H matrix is left constant. the private method `_fit_transform` is called by the exposed `non_negative_factorization` function.
In a scenario I've encountered, the user provides the H matrix, meaning the number of components is known a-prior, and there is no reason for the algorithm to run the lines
```
        if self._n_components is None:
            self._n_components = X.shape[1]
``` 
and raise an error later in the `_check_w_h`


https://github.com/scikit-learn/scikit-learn/blob/f5ec34e0f76277ba6d0a77d3033db0af83899b64/sklearn/decomposition/_nmf.py#LL1188C19-L1188C19

COMMENT:
It seems I'm unable to convey the scenario to you:
If you enter `update_H=False` then you do not initiate neither H nor W (which is just taken as the average entry of `X`
Only in that scenario the n_components should be ignored, as the decomposition rank is decided by the dimensions of the given H

Labels: help wanted, module:decomposition
NMF fit transform without updating H should not require the user to input "n_components"

The `_fit_transform` function of the `_nmf` module has the option to set `update_H=False`, where the H matrix is left constant. the private method `_fit_transform` is called by the exposed `non_negative_factorization` function.
In a scenario I've encountered, the user provides the H matrix, meaning the number of components is known a-prior, and there is no reason for the algorithm to run the lines
```
        if self._n_components is None:
            self._n_components = X.shape[1]
``` 
and raise an error later in the `_check_w_h`


https://github.com/scikit-learn/scikit-learn/blob/f5ec34e0f76277ba6d0a77d3033db0af83899b64/sklearn/decomposition/_nmf.py#LL1188C19-L1188C19

COMMENT:
Please provide a minimal reproducible, causing the error, and what you expect to happen. Would make it easier to investigate.

Labels: help wanted, module:decomposition
NMF fit transform without updating H should not require the user to input "n_components"

The `_fit_transform` function of the `_nmf` module has the option to set `update_H=False`, where the H matrix is left constant. the private method `_fit_transform` is called by the exposed `non_negative_factorization` function.
In a scenario I've encountered, the user provides the H matrix, meaning the number of components is known a-prior, and there is no reason for the algorithm to run the lines
```
        if self._n_components is None:
            self._n_components = X.shape[1]
``` 
and raise an error later in the `_check_w_h`


https://github.com/scikit-learn/scikit-learn/blob/f5ec34e0f76277ba6d0a77d3033db0af83899b64/sklearn/decomposition/_nmf.py#LL1188C19-L1188C19

COMMENT:
reproducible code:
```
import numpy as np
from sklearn.decomposition import non_negative_factorization

W_true = np.random.rand(6, 2)
H_true = np.random.rand(2, 5)
X = np.dot(W_true, H_true)

W, H, n_iter = non_negative_factorization(X, H=H_true, update_H=False)
```

I get the error: 
```
Traceback (most recent call last):
  File "/Applications/miniconda3/envs/gep-dynamics/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3460, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-14-a8ac745879a9>", line 1, in <module>
    W, H, n_iter = non_negative_factorization(X, H=H_true, update_H=False)
  File "/Applications/miniconda3/envs/gep-dynamics/lib/python3.9/site-packages/sklearn/utils/_param_validation.py", line 192, in wrapper
    return func(*args, **kwargs)
  File "/Applications/miniconda3/envs/gep-dynamics/lib/python3.9/site-packages/sklearn/decomposition/_nmf.py", line 1111, in non_negative_factorization
    W, H, n_iter = est._fit_transform(X, W=W, H=H, update_H=update_H)
  File "/Applications/miniconda3/envs/gep-dynamics/lib/python3.9/site-packages/sklearn/decomposition/_nmf.py", line 1625, in _fit_transform
    W, H = self._check_w_h(X, W, H, update_H)
  File "/Applications/miniconda3/envs/gep-dynamics/lib/python3.9/site-packages/sklearn/decomposition/_nmf.py", line 1184, in _check_w_h
    _check_init(H, (self._n_components, n_features), "NMF (input H)")
  File "/Applications/miniconda3/envs/gep-dynamics/lib/python3.9/site-packages/sklearn/decomposition/_nmf.py", line 68, in _check_init
    raise ValueError(
ValueError: Array with wrong shape passed to NMF (input H). Expected (5, 5), but got (2, 5) 
```

The error is caused due to the wrong dimensions of `self._n_components`. The source of the wrong dimension is that `_fit_transform` calls `self._check_params(X)`, which doesn't see a value assinged to `self.n_components` sets `self._n_components = X.shape[1]`. The error can be avoided by providing the `n_components` argument.

The key point of my issue is that when `H` is provided by the user, then **clearly** the user means to have `H.shape[0]` components in the decomposition, and thus the `n_components` argument is redundant.

Labels: help wanted, module:decomposition
NMF fit transform without updating H should not require the user to input "n_components"

The `_fit_transform` function of the `_nmf` module has the option to set `update_H=False`, where the H matrix is left constant. the private method `_fit_transform` is called by the exposed `non_negative_factorization` function.
In a scenario I've encountered, the user provides the H matrix, meaning the number of components is known a-prior, and there is no reason for the algorithm to run the lines
```
        if self._n_components is None:
            self._n_components = X.shape[1]
``` 
and raise an error later in the `_check_w_h`


https://github.com/scikit-learn/scikit-learn/blob/f5ec34e0f76277ba6d0a77d3033db0af83899b64/sklearn/decomposition/_nmf.py#LL1188C19-L1188C19

COMMENT:
As I said, the default ``n_components=None`` doesn't mean unspecified n_components, but automatically set ``n_components=n_features``. When H is user provided, there's a check for constistency between ``n_components`` and ``H.shape[0]``. I think this is a desirable behavior, rather than having ``n_components=None`` to mean a different thing based on ``update_H`` being True or False. What prevents you from setting ``n_components=H.shape[0]`` ?

Labels: help wanted, module:decomposition
NMF fit transform without updating H should not require the user to input "n_components"

The `_fit_transform` function of the `_nmf` module has the option to set `update_H=False`, where the H matrix is left constant. the private method `_fit_transform` is called by the exposed `non_negative_factorization` function.
In a scenario I've encountered, the user provides the H matrix, meaning the number of components is known a-prior, and there is no reason for the algorithm to run the lines
```
        if self._n_components is None:
            self._n_components = X.shape[1]
``` 
and raise an error later in the `_check_w_h`


https://github.com/scikit-learn/scikit-learn/blob/f5ec34e0f76277ba6d0a77d3033db0af83899b64/sklearn/decomposition/_nmf.py#LL1188C19-L1188C19

COMMENT:
Diving into the code, i now see the issue has nothing to do with `update_H`. If a user provides either `W` or `H`, then `n_components` should be set accordingly. This is a completely different scenario then when neither of the both is provided, and users shouldn't have the need to specify n_components

Labels: help wanted, module:decomposition
NMF fit transform without updating H should not require the user to input "n_components"

The `_fit_transform` function of the `_nmf` module has the option to set `update_H=False`, where the H matrix is left constant. the private method `_fit_transform` is called by the exposed `non_negative_factorization` function.
In a scenario I've encountered, the user provides the H matrix, meaning the number of components is known a-prior, and there is no reason for the algorithm to run the lines
```
        if self._n_components is None:
            self._n_components = X.shape[1]
``` 
and raise an error later in the `_check_w_h`


https://github.com/scikit-learn/scikit-learn/blob/f5ec34e0f76277ba6d0a77d3033db0af83899b64/sklearn/decomposition/_nmf.py#LL1188C19-L1188C19

COMMENT:
What prevents you from setting ``n_components=H.shape[0]`` ?

Labels: help wanted, module:decomposition
NMF fit transform without updating H should not require the user to input "n_components"

The `_fit_transform` function of the `_nmf` module has the option to set `update_H=False`, where the H matrix is left constant. the private method `_fit_transform` is called by the exposed `non_negative_factorization` function.
In a scenario I've encountered, the user provides the H matrix, meaning the number of components is known a-prior, and there is no reason for the algorithm to run the lines
```
        if self._n_components is None:
            self._n_components = X.shape[1]
``` 
and raise an error later in the `_check_w_h`


https://github.com/scikit-learn/scikit-learn/blob/f5ec34e0f76277ba6d0a77d3033db0af83899b64/sklearn/decomposition/_nmf.py#LL1188C19-L1188C19

COMMENT:
I personally don't believe in giving a function the same information twice, and errors that don't make sense until you dive into the classes where the functions are defined.
If a user gives the data of the rank (implicitly in the dimensions of W/H), why make them give the same information again explicitly?

Labels: help wanted, module:decomposition
NMF fit transform without updating H should not require the user to input "n_components"

The `_fit_transform` function of the `_nmf` module has the option to set `update_H=False`, where the H matrix is left constant. the private method `_fit_transform` is called by the exposed `non_negative_factorization` function.
In a scenario I've encountered, the user provides the H matrix, meaning the number of components is known a-prior, and there is no reason for the algorithm to run the lines
```
        if self._n_components is None:
            self._n_components = X.shape[1]
``` 
and raise an error later in the `_check_w_h`


https://github.com/scikit-learn/scikit-learn/blob/f5ec34e0f76277ba6d0a77d3033db0af83899b64/sklearn/decomposition/_nmf.py#LL1188C19-L1188C19

COMMENT:
> errors that don't make sense until you dive into the classes where the functions are defined.

The error makes sense because `n_components=None` is documented as equivalent to `n_components=n_features`. Then it is expected that an error is raised if `W` or `H` doesn't have the appropriate shape, since it does not correspond to the requested `n_components`.

I'm not against changing the default to `n_components="auto"` (with a deprecation cycle) such that:
- if neither W or H are provided, it defaults to `n_features`
- if `H` is provided, it's inferred from `H`
- if `W` and `H` are provided, it's inferred from both and if their shape don't match, an error is raised
- in any case, if n_components != "auto" and `W` or `H` is provided, an error is raised if they don't match.

Labels: help wanted, module:decomposition
SplineTransformer(extrapolate="periodic") outputs nan values for constant features

While reviewing #24145 I discovered the following bug:

```python
In [1]: import numpy as np

In [2]: from sklearn.preprocessing import SplineTransformer

In [3]: SplineTransformer(extrapolation="periodic").fit_transform(np.ones(shape=(5, 1)))
Out[3]: 
array([[nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan]])
```

Batman!

This is caused by:

- https://github.com/scikit-learn/scikit-learn/blob/ea046f024694b9a558c882b8c2610c52dad95e29/sklearn/preprocessing/_polynomial.py#L962-L971

`spl.t[n] - spl.t[spl.k]` is typically zero for constant features.

While fixing it for constant features is probably easy (just use `x = X[:, i]`), I wonder if other related numerical stability problems can be triggered for nearly constant features.

But maybe we can solve this problem in two stages: first the nan problem caused by modulus by exact zero and then investigate behavior on nearly constant data.

COMMENT:
I ran a quick check and it's indeed possible to get nans with near constant features:

```python
In [15]: SplineTransformer(extrapolation="periodic").fit_transform(np.ones(shape=(5, 1)) + np.random.randn(5, 1) * 1e-16)
Out[15]: 
array([[0.33333333, 0.66666667, 0.        , 0.        ],
       [0.33333333, 0.66666667, 0.        , 0.        ],
       [0.        , 0.        , 0.66666667, 0.33333333],
       [0.        , 0.        , 0.66666667, 0.33333333],
       [0.33333333, 0.66666667, 0.        , 0.        ]])

In [16]: SplineTransformer(extrapolation="periodic").fit_transform(np.ones(shape=(5, 1)) + np.random.randn(5, 1) * 1e-17)
Out[16]: 
array([[nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan]])
```

Labels: Bug