Talip7/scikit-learn-issues-multilabel huggingface dataset:
Estimator set_params API typo or misleading

### Describe the issue linked to the documentation

Not sure if the `set_params()` documentation is accurate.  [Relevant section of Estimator API documentation](https://scikit-learn.org/stable/developers/develop.html#get-params-and-set-params)

> The `set_params` on the other hand *takes as input a dict* of the form `'parameter': value` and sets the parameter of the estimator using this dict [emphasis added]

As far as I can tell, the subtle issue is that `set_params` takes keyword arguments, and **not a `dict`**.  It only receives a dict if it has splatted kwargs (`**kwargs`)

The wording led me to believe that the following method signature was acceptable:
```python
    def set_params(self, params: dict):
```
It gives `TypeError: set_params() got an unexpected keyword argument '<param name>'`

### Suggest a potential alternative/fix

> The `set_params` on the other hand takes as input each of the arguments of `__init__`, splats them into a dict of the form `'parameter': value`, and sets the parameter of the estimator using this dict

COMMENT:
I am happy with a PR to update the wording. For me, I find "splats" confusing here. In Python, I often see the `**` operator called "unpacking. So in this case:

> ... the arguments of `__init__`, unpacks them into a dict of the form `'parameter': value`

Labels: Documentation
Estimator set_params API typo or misleading

### Describe the issue linked to the documentation

Not sure if the `set_params()` documentation is accurate.  [Relevant section of Estimator API documentation](https://scikit-learn.org/stable/developers/develop.html#get-params-and-set-params)

> The `set_params` on the other hand *takes as input a dict* of the form `'parameter': value` and sets the parameter of the estimator using this dict [emphasis added]

As far as I can tell, the subtle issue is that `set_params` takes keyword arguments, and **not a `dict`**.  It only receives a dict if it has splatted kwargs (`**kwargs`)

The wording led me to believe that the following method signature was acceptable:
```python
    def set_params(self, params: dict):
```
It gives `TypeError: set_params() got an unexpected keyword argument '<param name>'`

### Suggest a potential alternative/fix

> The `set_params` on the other hand takes as input each of the arguments of `__init__`, splats them into a dict of the form `'parameter': value`, and sets the parameter of the estimator using this dict

COMMENT:
Working on this

Labels: Documentation
Inside SearchCV objects, enable `return_estimators`, similar to `cross_validate`

### Describe the workflow you want to enable

I am running GridSearch/RandomizedSearchCV and need to have access to the underlying estimators to perform some inquiries on the fitted objects based on their performances. However, there is no way of getting the fitted estimators.


### Describe your proposed solution

Add a `return_estimators=False` argument to the Grid/RandomizedSearchCV objects with a warning that one should use caution if their search space is large, since some models can be nontrivial size.

I would be interested in adding a PR if this is an acceptable feature request.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

This came up in context of trying to investigate tree sizes based on their optimal performances fitted using SearchCV. 

cc: @ogri

COMMENT:
Hi, I think you can use  grid_model.best_param to get the best estimator results.

Labels: New Feature, module:model_selection, Needs Decision - Include Feature
Inside SearchCV objects, enable `return_estimators`, similar to `cross_validate`

### Describe the workflow you want to enable

I am running GridSearch/RandomizedSearchCV and need to have access to the underlying estimators to perform some inquiries on the fitted objects based on their performances. However, there is no way of getting the fitted estimators.


### Describe your proposed solution

Add a `return_estimators=False` argument to the Grid/RandomizedSearchCV objects with a warning that one should use caution if their search space is large, since some models can be nontrivial size.

I would be interested in adding a PR if this is an acceptable feature request.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

This came up in context of trying to investigate tree sizes based on their optimal performances fitted using SearchCV. 

cc: @ogri

COMMENT:
Hi @reenam1998 that is not the goal. We would like to extract all the fitted estimators to determine properties of the fitted estimators compared to the array of scores. 

The `best_param` only gives one estimator corresponding to the `max(scores)`.

Labels: New Feature, module:model_selection, Needs Decision - Include Feature
Inside SearchCV objects, enable `return_estimators`, similar to `cross_validate`

### Describe the workflow you want to enable

I am running GridSearch/RandomizedSearchCV and need to have access to the underlying estimators to perform some inquiries on the fitted objects based on their performances. However, there is no way of getting the fitted estimators.


### Describe your proposed solution

Add a `return_estimators=False` argument to the Grid/RandomizedSearchCV objects with a warning that one should use caution if their search space is large, since some models can be nontrivial size.

I would be interested in adding a PR if this is an acceptable feature request.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

This came up in context of trying to investigate tree sizes based on their optimal performances fitted using SearchCV. 

cc: @ogri

COMMENT:
Hi @ogrisel I just wanted to see if you think this would be desirable? If so, I was planning on incorporating this asap so we can use it in our exps.

Labels: New Feature, module:model_selection, Needs Decision - Include Feature
Inside SearchCV objects, enable `return_estimators`, similar to `cross_validate`

### Describe the workflow you want to enable

I am running GridSearch/RandomizedSearchCV and need to have access to the underlying estimators to perform some inquiries on the fitted objects based on their performances. However, there is no way of getting the fitted estimators.


### Describe your proposed solution

Add a `return_estimators=False` argument to the Grid/RandomizedSearchCV objects with a warning that one should use caution if their search space is large, since some models can be nontrivial size.

I would be interested in adding a PR if this is an acceptable feature request.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

This came up in context of trying to investigate tree sizes based on their optimal performances fitted using SearchCV. 

cc: @ogri

COMMENT:
> This came up in context of trying to investigate tree sizes based on their optimal performances fitted using SearchCV.

Is there another use case for `return_estimators` besides looking at a model's size?

Labels: New Feature, module:model_selection, Needs Decision - Include Feature
Inside SearchCV objects, enable `return_estimators`, similar to `cross_validate`

### Describe the workflow you want to enable

I am running GridSearch/RandomizedSearchCV and need to have access to the underlying estimators to perform some inquiries on the fitted objects based on their performances. However, there is no way of getting the fitted estimators.


### Describe your proposed solution

Add a `return_estimators=False` argument to the Grid/RandomizedSearchCV objects with a warning that one should use caution if their search space is large, since some models can be nontrivial size.

I would be interested in adding a PR if this is an acceptable feature request.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

This came up in context of trying to investigate tree sizes based on their optimal performances fitted using SearchCV. 

cc: @ogri

COMMENT:
I would say investigating any fitted attributes. 

For example for trees we might be interested in the depth of the trees and the decision boundaries and then doing some meta analyses. 

For the size I think that could actually be a separate entry in the returned results in addition to what's already there. 

But I think returning estimators themselves serves additional use cases.

Labels: New Feature, module:model_selection, Needs Decision - Include Feature
BUG unpenalized GLM with LBFGS solver does not yield minimum norm solution with fit_intercept=True

### Summary
While unpenalized GLMs like `PoissonRegressor(alpha=0, fit_intercept=False)` yield a minimum norm solution on wide data, `PoissonRegressor(alpha=0, fit_intercept=True)` does not.

Detected in #23314.

#### Minimal Reproducible Example
```python
import numpy as np
from numpy.testing import assert_allclose
import pytest
from sklearn.datasets import make_low_rank_matrix
from sklearn.linear_model import PoissonRegressor
from sklearn.linear_model._linear_loss import LinearModelLoss


alpha = 0  # unpenalized, even tiny values like 1e-10 does not help.
fit_intercept = True  # if set to False, all tests pass.
model = PoissonRegressor(
    alpha = alpha,
    fit_intercept=fit_intercept,
    tol=1e-12,
    max_iter=1000,
)

# same as glm_dataset in test_glm.py for "wide"
n_samples, n_features = 4, 12
k = n_samples
rng = np.random.RandomState(42)
X = make_low_rank_matrix(
    n_samples=n_samples,
    n_features=n_features,
    effective_rank=k,
    tail_strength=0.1,
    random_state=rng,
)
X[:, -1] = 1  # last columns acts as intercept
U, s, Vt = np.linalg.svd(X, full_matrices=False)

raw_prediction = rng.uniform(low=-3, high=3, size=n_samples)
# minimum norm solution min ||w||_2 such that raw_prediction = X w:
# w = X'(XX')^-1 raw_prediction = V s^-1 U' raw_prediction
coef_unpenalized = Vt.T @ np.diag(1 / s) @ U.T @ raw_prediction

linear_loss = LinearModelLoss(base_loss=model._get_loss(), fit_intercept=True)
sw = np.full(shape=n_samples, fill_value=1 / n_samples)
y = linear_loss.base_loss.link.inverse(raw_prediction)

if fit_intercept:
    X = X[:, :-1]  # remove intercept
    intercept = coef_unpenalized[-1]
    coef = coef_unpenalized[:-1]
else:
    intercept = 0
    coef = coef_unpenalized

model.fit(X, y)

# This is true. So we have a solution.
assert_allclose(model.predict(X), y, rtol=1e-6)

# But it is NOT the minimum norm solution
norm_solution = np.linalg.norm(np.r_[intercept, coef])
norm_model = np.linalg.norm(np.r_[model.intercept_, model.coef_])
print(f"norm_solution = {norm_solution}")
print(f"norm_model    = {norm_model}")
assert norm_model == pytest.approx(norm_solution, rel=1e-4)
```
Output
```
norm_solution = 2.494710738642327
norm_model    = 2.5295744931923707
AssertionError
```

The strange thing is that setting `fit_intercept=False`, norms and coefficients are equal, i.e. the minimum norm solution is returned by the LBFGS solver.
On top, even setting a tiny positive penalty like `alpha=1e-10` does not give the minimum norm solution.

COMMENT:
I can use gradient clipping to avoid the above problems.

However this init is also slowing done convergence, in particular on Tweedie with p>=2 and identity link.

Labels: Bug, module:linear_model
BUG unpenalized GLM with LBFGS solver does not yield minimum norm solution with fit_intercept=True

### Summary
While unpenalized GLMs like `PoissonRegressor(alpha=0, fit_intercept=False)` yield a minimum norm solution on wide data, `PoissonRegressor(alpha=0, fit_intercept=True)` does not.

Detected in #23314.

#### Minimal Reproducible Example
```python
import numpy as np
from numpy.testing import assert_allclose
import pytest
from sklearn.datasets import make_low_rank_matrix
from sklearn.linear_model import PoissonRegressor
from sklearn.linear_model._linear_loss import LinearModelLoss


alpha = 0  # unpenalized, even tiny values like 1e-10 does not help.
fit_intercept = True  # if set to False, all tests pass.
model = PoissonRegressor(
    alpha = alpha,
    fit_intercept=fit_intercept,
    tol=1e-12,
    max_iter=1000,
)

# same as glm_dataset in test_glm.py for "wide"
n_samples, n_features = 4, 12
k = n_samples
rng = np.random.RandomState(42)
X = make_low_rank_matrix(
    n_samples=n_samples,
    n_features=n_features,
    effective_rank=k,
    tail_strength=0.1,
    random_state=rng,
)
X[:, -1] = 1  # last columns acts as intercept
U, s, Vt = np.linalg.svd(X, full_matrices=False)

raw_prediction = rng.uniform(low=-3, high=3, size=n_samples)
# minimum norm solution min ||w||_2 such that raw_prediction = X w:
# w = X'(XX')^-1 raw_prediction = V s^-1 U' raw_prediction
coef_unpenalized = Vt.T @ np.diag(1 / s) @ U.T @ raw_prediction

linear_loss = LinearModelLoss(base_loss=model._get_loss(), fit_intercept=True)
sw = np.full(shape=n_samples, fill_value=1 / n_samples)
y = linear_loss.base_loss.link.inverse(raw_prediction)

if fit_intercept:
    X = X[:, :-1]  # remove intercept
    intercept = coef_unpenalized[-1]
    coef = coef_unpenalized[:-1]
else:
    intercept = 0
    coef = coef_unpenalized

model.fit(X, y)

# This is true. So we have a solution.
assert_allclose(model.predict(X), y, rtol=1e-6)

# But it is NOT the minimum norm solution
norm_solution = np.linalg.norm(np.r_[intercept, coef])
norm_model = np.linalg.norm(np.r_[model.intercept_, model.coef_])
print(f"norm_solution = {norm_solution}")
print(f"norm_model    = {norm_model}")
assert norm_model == pytest.approx(norm_solution, rel=1e-4)
```
Output
```
norm_solution = 2.494710738642327
norm_model    = 2.5295744931923707
AssertionError
```

The strange thing is that setting `fit_intercept=False`, norms and coefficients are equal, i.e. the minimum norm solution is returned by the LBFGS solver.
On top, even setting a tiny positive penalty like `alpha=1e-10` does not give the minimum norm solution.

COMMENT:
https://fa.bianp.net/blog/2022/implicit-bias-regression/ even gives a nice proof.

Labels: Bug, module:linear_model
_svmlight_format_fast.pyx does not compile with Cython alpha version

The change introduced in https://github.com/scikit-learn/scikit-learn/pull/23127 do not compile with Cython alpha version.

To reproduce:
```
pip install --pre cython
cython sklearn/datasets/_svmlight_format_fast.pyx
```

Error:
```
Error compiling Cython file:
------------------------------------------------------------
...
    for k in range(row_length):
        val = X[row,k]
        if val == 0:
            continue
        x_inds[x_nz_used] = k
        x_vals[x_nz_used] = val
                           ^
------------------------------------------------------------

sklearn/datasets/_svmlight_format_fast.pyx:150:28: Cannot assign type 'double' to 'long long'
```

I am not quite sure why it was compiling before and what the proper fix is or even if this is a cython regression ...

I seem to remember that since x_vals and val are both cython fused types, cython will try to compile for the cross-product of possible types. Maybe there is a way to use http://docs.cython.org/en/latest/src/userguide/fusedtypes.html#type-checking-specializations or there may be a better way ...

The scipy-dev build where I saw this issue:
https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=43388&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf&t=eb5122d5-ab7e-5479-a8ce-245b4d64938b&l=924

COMMENT:
Might this be linked to https://github.com/cython/cython/pull/3617 which was introduced in Cython 3.0.0a6?

Labels: Bug
_svmlight_format_fast.pyx does not compile with Cython alpha version

The change introduced in https://github.com/scikit-learn/scikit-learn/pull/23127 do not compile with Cython alpha version.

To reproduce:
```
pip install --pre cython
cython sklearn/datasets/_svmlight_format_fast.pyx
```

Error:
```
Error compiling Cython file:
------------------------------------------------------------
...
    for k in range(row_length):
        val = X[row,k]
        if val == 0:
            continue
        x_inds[x_nz_used] = k
        x_vals[x_nz_used] = val
                           ^
------------------------------------------------------------

sklearn/datasets/_svmlight_format_fast.pyx:150:28: Cannot assign type 'double' to 'long long'
```

I am not quite sure why it was compiling before and what the proper fix is or even if this is a cython regression ...

I seem to remember that since x_vals and val are both cython fused types, cython will try to compile for the cross-product of possible types. Maybe there is a way to use http://docs.cython.org/en/latest/src/userguide/fusedtypes.html#type-checking-specializations or there may be a better way ...

The scipy-dev build where I saw this issue:
https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=43388&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf&t=eb5122d5-ab7e-5479-a8ce-245b4d64938b&l=924

COMMENT:
Closing thanks to #23669

Labels: Bug
BallTree.query returns inconsistent indices between scikit-learn versions 0.24.1 and 1.1.1

### Describe the bug

For the same data, `BallTree.query` can return different nearest neighbour indices if these neighbours are equally distanced from a point depending on scikit-learn version. Specifically, this can be observed in 0.24.1 vs 1.1.1.  

### Steps/Code to Reproduce

```
import numpy as np
from sklearn.neighbors import BallTree

np.random.seed(61)

X = np.random.randint(0, 3, size=(10,2))  # generated dataset

tree = BallTree(X, 4, 'hamming')
distances, indices = tree.query(X, 3, return_distance=True, dualtree=False, breadth_first=False, sort_results=True)

for i, (point, distances, indices) in enumerate(zip(X, distances, indices)):
    print(f'index {i}: datapoint {point} distances: {[round(dist,2) for dist in distances]} indices: {indices}')

```


### Expected Results

scikit-learn versions 0.24.1 and 1.1.1 would be expected to consistently show the same nearest neighbour indices whenever multiple neighbours are equally distanced from a data point

### Actual Results

scikit-learn v. 0.24.1:

<img width="549" alt="scikit-0 24 1" src="https://user-images.githubusercontent.com/21366061/174240375-d7215db5-6137-4ed6-a0d3-647d41761e19.png">

scikit-learn v. 1.1.1:

<img width="549" alt="scikit-1 1 1" src="https://user-images.githubusercontent.com/21366061/174240420-e4684662-26b1-4808-ae72-8e4ccff8e751.png">




### Versions

```shell
System:
  python: 3.9.7 (v3.9.7:1016ef3790, Aug 30 2021, 16:39:15) [Clang 6.0 (clang-600.0.57)]
executable: /Users/korigo/Downloads/scikit-0.24/bin/python3
  machine: macOS-10.16-x86_64-i386-64bit
Python dependencies:
     pip: 22.0.4
  setuptools: 62.1.0
   sklearn: 0.24.1
    numpy: 1.22.4
    scipy: 1.8.1
    Cython: None
    pandas: None
  matplotlib: None
    joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

System:
  python: 3.9.7 (v3.9.7:1016ef3790, Aug 30 2021, 16:39:15) [Clang 6.0 (clang-600.0.57)]
executable: /Users/korigo/Downloads/scikit-1.1.1/bin/python3
  machine: macOS-10.16-x86_64-i386-64bit
Python dependencies:
   sklearn: 1.1.1
     pip: 22.0.4
  setuptools: 62.1.0
    numpy: 1.22.4
    scipy: 1.8.1
    Cython: None
    pandas: None
  matplotlib: None
    joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
Thank you for opening the issue!

@jjerphan @jeremiedbb Did anything change with tie breaking in `BallTree` between 0.24.1 and 1.11.1?

Labels: Bug, module:neighbors
BallTree.query returns inconsistent indices between scikit-learn versions 0.24.1 and 1.1.1

### Describe the bug

For the same data, `BallTree.query` can return different nearest neighbour indices if these neighbours are equally distanced from a point depending on scikit-learn version. Specifically, this can be observed in 0.24.1 vs 1.1.1.  

### Steps/Code to Reproduce

```
import numpy as np
from sklearn.neighbors import BallTree

np.random.seed(61)

X = np.random.randint(0, 3, size=(10,2))  # generated dataset

tree = BallTree(X, 4, 'hamming')
distances, indices = tree.query(X, 3, return_distance=True, dualtree=False, breadth_first=False, sort_results=True)

for i, (point, distances, indices) in enumerate(zip(X, distances, indices)):
    print(f'index {i}: datapoint {point} distances: {[round(dist,2) for dist in distances]} indices: {indices}')

```


### Expected Results

scikit-learn versions 0.24.1 and 1.1.1 would be expected to consistently show the same nearest neighbour indices whenever multiple neighbours are equally distanced from a data point

### Actual Results

scikit-learn v. 0.24.1:

<img width="549" alt="scikit-0 24 1" src="https://user-images.githubusercontent.com/21366061/174240375-d7215db5-6137-4ed6-a0d3-647d41761e19.png">

scikit-learn v. 1.1.1:

<img width="549" alt="scikit-1 1 1" src="https://user-images.githubusercontent.com/21366061/174240420-e4684662-26b1-4808-ae72-8e4ccff8e751.png">




### Versions

```shell
System:
  python: 3.9.7 (v3.9.7:1016ef3790, Aug 30 2021, 16:39:15) [Clang 6.0 (clang-600.0.57)]
executable: /Users/korigo/Downloads/scikit-0.24/bin/python3
  machine: macOS-10.16-x86_64-i386-64bit
Python dependencies:
     pip: 22.0.4
  setuptools: 62.1.0
   sklearn: 0.24.1
    numpy: 1.22.4
    scipy: 1.8.1
    Cython: None
    pandas: None
  matplotlib: None
    joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

System:
  python: 3.9.7 (v3.9.7:1016ef3790, Aug 30 2021, 16:39:15) [Clang 6.0 (clang-600.0.57)]
executable: /Users/korigo/Downloads/scikit-1.1.1/bin/python3
  machine: macOS-10.16-x86_64-i386-64bit
Python dependencies:
   sklearn: 1.1.1
     pip: 22.0.4
  setuptools: 62.1.0
    numpy: 1.22.4
    scipy: 1.8.1
    Cython: None
    pandas: None
  matplotlib: None
    joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
#19473 (which first appeared in 1.0.rc1) introduced this regression.

Labels: Bug, module:neighbors