Talip7/scikit-learn-issues-multilabel huggingface dataset:
Add gradient clipping to SGDRegressor for stability and differential privacy

### Describe the workflow you want to enable

Add gradient clipping to SGDRegressor to:
1. Improve training stability when dealing with outliers or ill-conditioned data
2. Enable differentially private regression by bounding the influence of any single observation

This addition would enable users to:
- Train more stable models on real-world data with outliers
- Implement differentially private regression with minimal additional code
- Control the maximum influence of any single observation on the model

### Describe your proposed solution

Add a `clip_value` parameter to SGDRegressor that bounds the L2 norm of gradients during training:

```python
# Example usage
model = SGDRegressor(clip_value=1.0)
```

Implementation would involve:
1. Add `clip_value` parameter (default: None for no clipping)
2. Add gradient clipping in the update step:
```python
def _clip_gradient(self, grad):
    if self.clip_value is None:
        return grad
    norm = np.linalg.norm(grad)
    if norm > self.clip_value:
        return grad * (self.clip_value / norm)
    return grad
```

### Research basis and precedent

Gradient clipping is well-established:
- Introduced in ["On the difficulty of training Recurrent Neural Networks"](https://arxiv.org/abs/1211.5063) (2013, [7000+ citations](https://scholar.google.com/scholar?cites=3353056030101542547&as_sdt=20005&sciodt=0,9&hl=en))
- Standard practice in deep learning frameworks (PyTorch, TensorFlow)
- Core component of differentially private SGD as formalized in ["Deep Learning with Differential Privacy"](https://arxiv.org/abs/1607.00133) (2016, [6000+ citations](https://scholar.google.com/scholar?cites=11431158613977668861&as_sdt=20005&sciodt=0,9&hl=en))

Implemented in:
- PyTorch: `torch.nn.utils.clip_grad_norm_`
- TensorFlow: `tf.clip_by_norm`
- JAX: `jax.example_libraries.optimizers.clip_grads`
- Scikit-learn's neural networks

### Benefits

1. **Stability**: Prevents exploding gradients and improves convergence on ill-conditioned problems
2. **Privacy**: Essential building block for differentially private regression
3. **Simple implementation**: Minimal code changes required
4. **No new hyperparameters** when using default None (maintains backward compatibility)

### Benchmarks

Would be happy to provide benchmarks showing:
1. Improved stability on datasets with outliers
2. Convergence comparisons with/without clipping
3. Privacy-utility tradeoffs when used for differential privacy

Let me know if you'd like me to provide implementation details or benchmarks to support this addition.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@john-zhang-uoft we don't really maintain the MLP class much, so adding new features like this to it would be out of scope.

Regarding the SGD case, cc @scikit-learn/core-devs . I'm okay with adding it since it's a very minimal change, but conceptually, it's quite different from the other things we've included in the project.

Labels: New Feature, Needs Decision
Add gradient clipping to SGDRegressor for stability and differential privacy

### Describe the workflow you want to enable

Add gradient clipping to SGDRegressor to:
1. Improve training stability when dealing with outliers or ill-conditioned data
2. Enable differentially private regression by bounding the influence of any single observation

This addition would enable users to:
- Train more stable models on real-world data with outliers
- Implement differentially private regression with minimal additional code
- Control the maximum influence of any single observation on the model

### Describe your proposed solution

Add a `clip_value` parameter to SGDRegressor that bounds the L2 norm of gradients during training:

```python
# Example usage
model = SGDRegressor(clip_value=1.0)
```

Implementation would involve:
1. Add `clip_value` parameter (default: None for no clipping)
2. Add gradient clipping in the update step:
```python
def _clip_gradient(self, grad):
    if self.clip_value is None:
        return grad
    norm = np.linalg.norm(grad)
    if norm > self.clip_value:
        return grad * (self.clip_value / norm)
    return grad
```

### Research basis and precedent

Gradient clipping is well-established:
- Introduced in ["On the difficulty of training Recurrent Neural Networks"](https://arxiv.org/abs/1211.5063) (2013, [7000+ citations](https://scholar.google.com/scholar?cites=3353056030101542547&as_sdt=20005&sciodt=0,9&hl=en))
- Standard practice in deep learning frameworks (PyTorch, TensorFlow)
- Core component of differentially private SGD as formalized in ["Deep Learning with Differential Privacy"](https://arxiv.org/abs/1607.00133) (2016, [6000+ citations](https://scholar.google.com/scholar?cites=11431158613977668861&as_sdt=20005&sciodt=0,9&hl=en))

Implemented in:
- PyTorch: `torch.nn.utils.clip_grad_norm_`
- TensorFlow: `tf.clip_by_norm`
- JAX: `jax.example_libraries.optimizers.clip_grads`
- Scikit-learn's neural networks

### Benefits

1. **Stability**: Prevents exploding gradients and improves convergence on ill-conditioned problems
2. **Privacy**: Essential building block for differentially private regression
3. **Simple implementation**: Minimal code changes required
4. **No new hyperparameters** when using default None (maintains backward compatibility)

### Benchmarks

Would be happy to provide benchmarks showing:
1. Improved stability on datasets with outliers
2. Convergence comparisons with/without clipping
3. Privacy-utility tradeoffs when used for differential privacy

Let me know if you'd like me to provide implementation details or benchmarks to support this addition.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I'm pretty sure the gradient clipping that was requested by this issue is slightly different than what is currently done (i.e. clipping the gradient with respect to weights instead of with respect to the output) and my PR shows this. Once again this is my very first time trying to contribute to open source, so feel free to leave any feedback.

Labels: New Feature, Needs Decision
Reduce redundancy in floating type checks for Array API support

### Describe the workflow you want to enable

While working on #29978, we noticed that the following procedure is repeated across most regression metrics in `_regression.py` for the Array API:

```python
    xp, _ = get_namespace(y_true, y_pred, sample_weight, multioutput)
    dtype = _find_matching_floating_dtype(y_true, y_pred, xp=xp)

    _, y_true, y_pred, multioutput = _check_reg_targets(
        y_true, y_pred, multioutput, dtype=dtype, xp=xp
    )
```

To reduce redundancy, it would make sense to incorporate the `_find_matching_floating_dtype` logic directly into the `_check_reg_targets` function. This would result in the following cleaner implementation:

```python
    xp, _ = get_namespace(y_true, y_pred, sample_weight, multioutput)
    _, y_true, y_pred, multioutput, dtype = _check_reg_targets(
        y_true, y_pred, multioutput, xp=xp
    )
```

### Describe your proposed solution

We could introduce a new function, `_check_reg_targets_and_dtype`, defined in the obvious way. This approach would enable us to utilise the existing tests in `test_regression.py` with minimal changes.

### Describe alternatives you've considered, if relevant

We could modify the original `_check_reg_targets` function, but this would require carefully reviewing and updating the relevant tests in `test_regression.py` to ensure everything remains consistent.

### Additional context

This is part of the Array API project #26024.

ping: @ogrisel
cc: @glemaitre, @betatim, @adrinjalali.

COMMENT:
That sounds good to me.

Labels: New Feature, Array API
Reduce redundancy in floating type checks for Array API support

### Describe the workflow you want to enable

While working on #29978, we noticed that the following procedure is repeated across most regression metrics in `_regression.py` for the Array API:

```python
    xp, _ = get_namespace(y_true, y_pred, sample_weight, multioutput)
    dtype = _find_matching_floating_dtype(y_true, y_pred, xp=xp)

    _, y_true, y_pred, multioutput = _check_reg_targets(
        y_true, y_pred, multioutput, dtype=dtype, xp=xp
    )
```

To reduce redundancy, it would make sense to incorporate the `_find_matching_floating_dtype` logic directly into the `_check_reg_targets` function. This would result in the following cleaner implementation:

```python
    xp, _ = get_namespace(y_true, y_pred, sample_weight, multioutput)
    _, y_true, y_pred, multioutput, dtype = _check_reg_targets(
        y_true, y_pred, multioutput, xp=xp
    )
```

### Describe your proposed solution

We could introduce a new function, `_check_reg_targets_and_dtype`, defined in the obvious way. This approach would enable us to utilise the existing tests in `test_regression.py` with minimal changes.

### Describe alternatives you've considered, if relevant

We could modify the original `_check_reg_targets` function, but this would require carefully reviewing and updating the relevant tests in `test_regression.py` to ensure everything remains consistent.

### Additional context

This is part of the Array API project #26024.

ping: @ogrisel
cc: @glemaitre, @betatim, @adrinjalali.

COMMENT:
Hi @virchan , @adrinjalali 

Will it be okay if I try to work on this?

Labels: New Feature, Array API
Reduce redundancy in floating type checks for Array API support

### Describe the workflow you want to enable

While working on #29978, we noticed that the following procedure is repeated across most regression metrics in `_regression.py` for the Array API:

```python
    xp, _ = get_namespace(y_true, y_pred, sample_weight, multioutput)
    dtype = _find_matching_floating_dtype(y_true, y_pred, xp=xp)

    _, y_true, y_pred, multioutput = _check_reg_targets(
        y_true, y_pred, multioutput, dtype=dtype, xp=xp
    )
```

To reduce redundancy, it would make sense to incorporate the `_find_matching_floating_dtype` logic directly into the `_check_reg_targets` function. This would result in the following cleaner implementation:

```python
    xp, _ = get_namespace(y_true, y_pred, sample_weight, multioutput)
    _, y_true, y_pred, multioutput, dtype = _check_reg_targets(
        y_true, y_pred, multioutput, xp=xp
    )
```

### Describe your proposed solution

We could introduce a new function, `_check_reg_targets_and_dtype`, defined in the obvious way. This approach would enable us to utilise the existing tests in `test_regression.py` with minimal changes.

### Describe alternatives you've considered, if relevant

We could modify the original `_check_reg_targets` function, but this would require carefully reviewing and updating the relevant tests in `test_regression.py` to ensure everything remains consistent.

### Additional context

This is part of the Array API project #26024.

ping: @ogrisel
cc: @glemaitre, @betatim, @adrinjalali.

COMMENT:
> Hi @virchan , @adrinjalali
> 
> Will it be okay if I try to work on this?

Hello @sqali, 

This issue is currently affecting another PR I'm working on, so I feel obligated to resolve it myself. I plan to start working on it once the proposed solution is approved.

That said, your review of the PR would still be very much appreciated to catch any mistakes or suggest improvements. I'll make sure to CC you when the PR is created.

Labels: New Feature, Array API
Reduce redundancy in floating type checks for Array API support

### Describe the workflow you want to enable

While working on #29978, we noticed that the following procedure is repeated across most regression metrics in `_regression.py` for the Array API:

```python
    xp, _ = get_namespace(y_true, y_pred, sample_weight, multioutput)
    dtype = _find_matching_floating_dtype(y_true, y_pred, xp=xp)

    _, y_true, y_pred, multioutput = _check_reg_targets(
        y_true, y_pred, multioutput, dtype=dtype, xp=xp
    )
```

To reduce redundancy, it would make sense to incorporate the `_find_matching_floating_dtype` logic directly into the `_check_reg_targets` function. This would result in the following cleaner implementation:

```python
    xp, _ = get_namespace(y_true, y_pred, sample_weight, multioutput)
    _, y_true, y_pred, multioutput, dtype = _check_reg_targets(
        y_true, y_pred, multioutput, xp=xp
    )
```

### Describe your proposed solution

We could introduce a new function, `_check_reg_targets_and_dtype`, defined in the obvious way. This approach would enable us to utilise the existing tests in `test_regression.py` with minimal changes.

### Describe alternatives you've considered, if relevant

We could modify the original `_check_reg_targets` function, but this would require carefully reviewing and updating the relevant tests in `test_regression.py` to ensure everything remains consistent.

### Additional context

This is part of the Array API project #26024.

ping: @ogrisel
cc: @glemaitre, @betatim, @adrinjalali.

COMMENT:
Thanks @virchan

Labels: New Feature, Array API
Inconsistency between lars_path documentation and behavior in code

### Describe the issue linked to the documentation

While using the `lars_path` function from the `sklearn.linear_model` module, I came across a confusing behavior that seems to contradict the documentation.

According to the [documentation for `lars_path`](https://scikit-learn.org/dev/modules/generated/sklearn.linear_model.lars_path.html), it states that:

> X : None or ndarray of shape (n_samples, n_features)
Input data. Note that if X is None then the Gram matrix must be specified, i.e, cannot be None or False .

However, when I passed `X=None` and provided the `Gram` matrix, I encountered the following error in the code:

```python
if X is None and Gram is not None:
    raise ValueError("X cannot be None if Gram is not None. Use lars_path_gram to avoid passing X and y.")
```

This directly contradicts what the documentation suggests, as I expected lars_path to work with X=None as long as the Gram matrix was given, but instead, I got a ValueError.

Could you help to look into this issue?
Thank you for the attention!

### Suggest a potential alternative/fix

I would suggest to update the documentation to align with the current behavior of the code.

COMMENT:
Making a bit of archeology, it seems that the documentation is the culprit:

https://github.com/scikit-learn/scikit-learn/pull/11699/files#diff-37f25c73e0ef2f228cbd8edbb6904ef893b77caf42256a27ff0714adb196e4a8R46-R54

It seems that we deprecated the behaviour many years ago and advocated for the usage of `lars_path_gram` instead. So, the documentation needs to be updated but not the code.

@xie00059 do you want to make a PR to correct this error.

Labels: Documentation
`from sklearn import this`

### Describe the workflow you want to enable

It's not just Python, there are also a lot of cool packages that [import this](https://calmcode.io/til/python-import-this). It something that I have taken to heart personally on many of my own open-source packages but it also seems that the Narwhals project has started to add a poem containing the lessons learned. 

![image](https://github.com/user-attachments/assets/4e7dcf8b-8b6d-4bce-b73b-a83671d8e339)

Maybe it would be nice to share the lessons learned while maintaining this lesson in a poem that is part of the package? I would certainly be interested in it!

### Describe your proposed solution

I am not at all a frequent committer here, but I can imagine that it might be a fun community project to see if we can come up with a poem that best represents this project. I am open to many methods of going about it though because I can certainly see that it can be done in many ways. It might even be a fun team-building exercise for the core team?

### Describe alternatives you've considered, if relevant

I had a quick stab at a short poem myself, but am open to suggestions on how to expand it ;) 

```
Care about the code, every single bit.
Designing a clear API makes it much easier to fit.
When things can break, you want to be strict. 
Keep your mind open, the future is very hard to predict. 
```

### Additional context

_No response_

COMMENT:
I'd be happy to see this, but personally I'm the worst at coming up with rhymes lol.

It'd be a fun exercise for us all to try to come up with something to be added to the text, for people who fancy doing it.

cc @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team

Labels: RFC
`from sklearn import this`

### Describe the workflow you want to enable

It's not just Python, there are also a lot of cool packages that [import this](https://calmcode.io/til/python-import-this). It something that I have taken to heart personally on many of my own open-source packages but it also seems that the Narwhals project has started to add a poem containing the lessons learned. 

![image](https://github.com/user-attachments/assets/4e7dcf8b-8b6d-4bce-b73b-a83671d8e339)

Maybe it would be nice to share the lessons learned while maintaining this lesson in a poem that is part of the package? I would certainly be interested in it!

### Describe your proposed solution

I am not at all a frequent committer here, but I can imagine that it might be a fun community project to see if we can come up with a poem that best represents this project. I am open to many methods of going about it though because I can certainly see that it can be done in many ways. It might even be a fun team-building exercise for the core team?

### Describe alternatives you've considered, if relevant

I had a quick stab at a short poem myself, but am open to suggestions on how to expand it ;) 

```
Care about the code, every single bit.
Designing a clear API makes it much easier to fit.
When things can break, you want to be strict. 
Keep your mind open, the future is very hard to predict. 
```

### Additional context

_No response_

COMMENT:
@koaning I like your idea and your poem! :) 

But you're asking us to match your and  @MarcoGorelli 's level of creativity? The challenge is on! 😅

Labels: RFC
svcmodel.fit(X_train,y_train) on GPU? we need native GPU mode for scikit-learn

### Describe the workflow you want to enable

svcmodel.fit(X_train,y_train) on GPU?

we need native GPU mode for scikit-learn

### Describe your proposed solution

svcmodel.fit(X_train,y_train) on GPU?

we need native GPU mode for scikit-learn

### Describe alternatives you've considered, if relevant

svcmodel.fit(X_train,y_train) on GPU?

we need native GPU mode for scikit-learn

### Additional context

_No response_

COMMENT:
We use libsvm for SVM and right now this is out of scope for us since we can't use array API for this purpose.

cc @betatim @OmarManzoor @ogrisel in case they feel differently.

Labels: New Feature, Needs Triage
Error on the scikit-learn algorithm cheat-sheet?

### Describe the bug

In Clustering, if there are <10K samples, shouldn't yes go to Tough Luck (because there aren't enough samples), and no, go to MeanShift/VBGMM (because there are)?

### Steps/Code to Reproduce

# N/A

### Expected Results

# N/A

### Actual Results

# N/A

### Versions

```shell
# N/A
```

COMMENT:
I think you're right (although I personally think 10k is a very large number and with low number of features clustering couple 100 samples also makes sense). Feel free to submit a pull request with the fix.

Labels: Documentation
Error on the scikit-learn algorithm cheat-sheet?

### Describe the bug

In Clustering, if there are <10K samples, shouldn't yes go to Tough Luck (because there aren't enough samples), and no, go to MeanShift/VBGMM (because there are)?

### Steps/Code to Reproduce

# N/A

### Expected Results

# N/A

### Actual Results

# N/A

### Versions

```shell
# N/A
```

COMMENT:
cc @Charlie-XIAO maybe.

Labels: Documentation
Error on the scikit-learn algorithm cheat-sheet?

### Describe the bug

In Clustering, if there are <10K samples, shouldn't yes go to Tough Luck (because there aren't enough samples), and no, go to MeanShift/VBGMM (because there are)?

### Steps/Code to Reproduce

# N/A

### Expected Results

# N/A

### Actual Results

# N/A

### Versions

```shell
# N/A
```

COMMENT:
Indeed that's my bad. Fixing that SVG is a bit complicated, maybe I'll do that directly.

Labels: Documentation
Error on the scikit-learn algorithm cheat-sheet?

### Describe the bug

In Clustering, if there are <10K samples, shouldn't yes go to Tough Luck (because there aren't enough samples), and no, go to MeanShift/VBGMM (because there are)?

### Steps/Code to Reproduce

# N/A

### Expected Results

# N/A

### Actual Results

# N/A

### Versions

```shell
# N/A
```

COMMENT:
Well I double checked and saw that previous "no" also points to tough luck: https://scikit-learn.org/1.4/tutorial/machine_learning_map/index.html so it's actually not my typo. I'm thinking maybe it means that MeanShift and Variational BGM models are not suitable for large number of samples? (I'm not so familiar with those algorithms though.) @adrinjalali

On second thought I actually don't think <10K samples would be tough luck... Looking at the starting point of the graph there is actually an arrow pointing to "get more data" when we have less than 50 samples.

Labels: Documentation
Error on the scikit-learn algorithm cheat-sheet?

### Describe the bug

In Clustering, if there are <10K samples, shouldn't yes go to Tough Luck (because there aren't enough samples), and no, go to MeanShift/VBGMM (because there are)?

### Steps/Code to Reproduce

# N/A

### Expected Results

# N/A

### Actual Results

# N/A

### Versions

```shell
# N/A
```

COMMENT:
Oh that might be true. We'd need to check the implementation and see if that's true. Also, it might be the case that with the new hardware we have, the number threshold is quite a bit higher.

Labels: Documentation
Error on the scikit-learn algorithm cheat-sheet?

### Describe the bug

In Clustering, if there are <10K samples, shouldn't yes go to Tough Luck (because there aren't enough samples), and no, go to MeanShift/VBGMM (because there are)?

### Steps/Code to Reproduce

# N/A

### Expected Results

# N/A

### Actual Results

# N/A

### Versions

```shell
# N/A
```

COMMENT:
TL;DR: I believe the [cheat sheet](https://scikit-learn.org/1.5/machine_learning_map.html) is correct.

Suppose we are working with an unlabelled dataset with an unknown number of clusters.

For `sample_size >= 10K`, most scikit-learn clustering algorithms become computationally expensive, so the "tough luck" conclusion seems reasonable. I believe this is what the original author intended.

In the case of `50 < sample_size < 10K`, I don't see any issues with using scikit-learn’s `MeanShift` or `VBGMM`. Both algorithms are designed for such scenarios. At worst, they can serve as brute-force solutions if no better options are available.

Labels: Documentation
Error on the scikit-learn algorithm cheat-sheet?

### Describe the bug

In Clustering, if there are <10K samples, shouldn't yes go to Tough Luck (because there aren't enough samples), and no, go to MeanShift/VBGMM (because there are)?

### Steps/Code to Reproduce

# N/A

### Expected Results

# N/A

### Actual Results

# N/A

### Versions

```shell
# N/A
```

COMMENT:
Thanks everyone for the discussions, closing.

Labels: Documentation
Add TQDM progress bar to .fit

### Describe the workflow you want to enable

Add TQDM progress bar to .fit 

```
from sklearn.svm import SVC
svcmodel.fit(X_train,y_train)
```

### Describe your proposed solution

Add TQDM progress bar to .fit 

### Describe alternatives you've considered, if relevant

Add TQDM progress bar to .fit 

### Additional context

_No response_

COMMENT:
Duplicate of https://github.com/scikit-learn/scikit-learn/issues/7574

Labels: New Feature, Needs Triage