Talip7/scikit-learn-issues-multilabel huggingface dataset:
Verbosity option is not working in GridSearchCV (Jupyter notebook)

### Describe the bug

So this issue has been addressed before [here ](https://github.com/scikit-learn/scikit-learn/issues/22291) by [darrencl](https://github.com/darrencl), but the user didn't follow up with [lesteve](https://github.com/lesteve) response.

The problem is that GridSearchCV doesn't show the elapsed time periodically, or any log, I am set`n_jobs = -1`, and `verbose = 1`. I tried setting `n_jobs` to other values, the same with `verbose`, but nothing happened.
Note that this didn't happen until I updated scikit-learn from version 0.22.1 to 1.0.2.

[lesteve](https://github.com/lesteve) in his response assumed that this problem is due to `ipykernel <6`, which is not the case with me.

### Steps/Code to Reproduce

```
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn import datasets

iris = datasets.load_iris()
params = {'n_estimators':[10,20,30,40,50,60],
          'max_depth':[20,50,60,70,80]}
grid_obj = GridSearchCV(estimator=RandomForestClassifier(), param_grid=params, n_jobs=-1, verbose=1, cv=5)
grid_obj.fit(iris.data, iris.target)
```

### Expected Results

This is the output when using version 0.22.1
![image](https://user-images.githubusercontent.com/17684093/158401635-31755363-2108-41e7-8f7f-08985abd03b8.png)


### Actual Results

```Fitting 5 folds for each of 30 candidates, totalling 150 fits```

### Versions

```shell
System:
    python: 3.8.5 (default, Sep  3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)]
executable: D:\Programs\ApplicationsSetup\anaconda3\python.exe
   machine: Windows-10-10.0.19041-SP0

Python dependencies:
          pip: 21.2.2
   setuptools: 58.0.4
      sklearn: 1.0.2
        numpy: 1.19.2
        scipy: 1.6.2
       Cython: 0.29.25
       pandas: 1.4.1
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 2.2.0


!jupyter --version
Selected Jupyter core packages...
IPython          : 7.31.1
ipykernel        : 6.4.1
ipywidgets       : 7.6.5
jupyter_client   : 6.1.12
jupyter_core     : 4.9.1
jupyter_server   : 1.13.5
jupyterlab       : 3.2.9
nbclient         : 0.5.11
nbconvert        : 6.1.0
nbformat         : 5.1.3
notebook         : 6.4.9
qtconsole        : 5.2.2
traitlets        : 5.1.1
```

COMMENT:
Yes, please start by opening a separate issue for the regular command line execution on Linux, which definitely seems unexpected. Please add a standalone snippet reproducing the problem, which will make it a lot more likely it get looked at and eventually fixed.

On the SLURM one, it may be that there is some buffering going on in different places (SLURM side, OS side, ...), see https://github.com/dask/dask-jobqueue/issues/299#issuecomment-609002077 for some quick experiments I made a while ago on a SGE cluster.

Labels: Bug, module:model_selection, Needs Investigation
Verbosity option is not working in GridSearchCV (Jupyter notebook)

### Describe the bug

So this issue has been addressed before [here ](https://github.com/scikit-learn/scikit-learn/issues/22291) by [darrencl](https://github.com/darrencl), but the user didn't follow up with [lesteve](https://github.com/lesteve) response.

The problem is that GridSearchCV doesn't show the elapsed time periodically, or any log, I am set`n_jobs = -1`, and `verbose = 1`. I tried setting `n_jobs` to other values, the same with `verbose`, but nothing happened.
Note that this didn't happen until I updated scikit-learn from version 0.22.1 to 1.0.2.

[lesteve](https://github.com/lesteve) in his response assumed that this problem is due to `ipykernel <6`, which is not the case with me.

### Steps/Code to Reproduce

```
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn import datasets

iris = datasets.load_iris()
params = {'n_estimators':[10,20,30,40,50,60],
          'max_depth':[20,50,60,70,80]}
grid_obj = GridSearchCV(estimator=RandomForestClassifier(), param_grid=params, n_jobs=-1, verbose=1, cv=5)
grid_obj.fit(iris.data, iris.target)
```

### Expected Results

This is the output when using version 0.22.1
![image](https://user-images.githubusercontent.com/17684093/158401635-31755363-2108-41e7-8f7f-08985abd03b8.png)


### Actual Results

```Fitting 5 folds for each of 30 candidates, totalling 150 fits```

### Versions

```shell
System:
    python: 3.8.5 (default, Sep  3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)]
executable: D:\Programs\ApplicationsSetup\anaconda3\python.exe
   machine: Windows-10-10.0.19041-SP0

Python dependencies:
          pip: 21.2.2
   setuptools: 58.0.4
      sklearn: 1.0.2
        numpy: 1.19.2
        scipy: 1.6.2
       Cython: 0.29.25
       pandas: 1.4.1
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 2.2.0


!jupyter --version
Selected Jupyter core packages...
IPython          : 7.31.1
ipykernel        : 6.4.1
ipywidgets       : 7.6.5
jupyter_client   : 6.1.12
jupyter_core     : 4.9.1
jupyter_server   : 1.13.5
jupyterlab       : 3.2.9
nbclient         : 0.5.11
nbconvert        : 6.1.0
nbformat         : 5.1.3
notebook         : 6.4.9
qtconsole        : 5.2.2
traitlets        : 5.1.1
```

COMMENT:
@lesteve I'm also having the problem on SLURM as well, even after updating scikit-learn to 1.3.0. Only works properly when n_jobs=1. 

I also tried your suggestion in the comment you referenced above, but it didn't help. I'm not entirely sure if I did it right. I used the argument `--signal=B:TERM@n_duration` to send the SIGTERM signal `n_duration` seconds before the job's time limit is reached. Then I ran `sys.stdout.flush()` and `sys.stderr.flush()` in a function that I passed to `signal.signal(signal.SIGTERM, flush_function)`. It didn't resolve the issue.

Labels: Bug, module:model_selection, Needs Investigation
KFold splitter falsely claims it supports groups

@thomasjpfan : (taken from https://github.com/scikit-learn/scikit-learn/pull/22765/files#r825487547)

> My mind is a little blown. So the current API for KFold.split has a groups parameter it never uses. Because of the inheritance structure, the [API docs for KFold.split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html#sklearn.model_selection.KFold.split) says the group labels are used in KFold.
> 
> (This comment is not actionable)


I haven't looked at how to proceed here. It certainly shouldn't claim it supports `groups` in the docs, and we need to check how to fix the API if we can.

COMMENT:
The fix would involve either dynamically rewriting the docstring of the parent class or overriding `split` in subclasses that do not use `groups` and update the docstring.

Labels: Documentation, Moderate, module:model_selection
KFold splitter falsely claims it supports groups

@thomasjpfan : (taken from https://github.com/scikit-learn/scikit-learn/pull/22765/files#r825487547)

> My mind is a little blown. So the current API for KFold.split has a groups parameter it never uses. Because of the inheritance structure, the [API docs for KFold.split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html#sklearn.model_selection.KFold.split) says the group labels are used in KFold.
> 
> (This comment is not actionable)


I haven't looked at how to proceed here. It certainly shouldn't claim it supports `groups` in the docs, and we need to check how to fix the API if we can.

COMMENT:
The second option seems more like our usual practices :)

Labels: Documentation, Moderate, module:model_selection
KFold splitter falsely claims it supports groups

@thomasjpfan : (taken from https://github.com/scikit-learn/scikit-learn/pull/22765/files#r825487547)

> My mind is a little blown. So the current API for KFold.split has a groups parameter it never uses. Because of the inheritance structure, the [API docs for KFold.split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html#sklearn.model_selection.KFold.split) says the group labels are used in KFold.
> 
> (This comment is not actionable)


I haven't looked at how to proceed here. It certainly shouldn't claim it supports `groups` in the docs, and we need to check how to fix the API if we can.

COMMENT:
> The function signature should go from split(self, X, y=None, groups=None) to split(self, X, y=None).

No, we can not change the signature since it will break backward compatibility. `KFold` should have the same signature, but the docstring for `groups` changes:

```python
class KFold(...):
    def split(self, X, y=None, groups=None):
        """...

        groups : array-like of shape (n_samples,), default=None
            Not used, present for API consistency.
        ...
        """
        super().split(self, X, y=y)
```

Labels: Documentation, Moderate, module:model_selection
KFold splitter falsely claims it supports groups

@thomasjpfan : (taken from https://github.com/scikit-learn/scikit-learn/pull/22765/files#r825487547)

> My mind is a little blown. So the current API for KFold.split has a groups parameter it never uses. Because of the inheritance structure, the [API docs for KFold.split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html#sklearn.model_selection.KFold.split) says the group labels are used in KFold.
> 
> (This comment is not actionable)


I haven't looked at how to proceed here. It certainly shouldn't claim it supports `groups` in the docs, and we need to check how to fix the API if we can.

COMMENT:
Makes sense.

As far as I can tell, the following cross-validators should have their docstring for the `groups` argument changed.

- `KFold`

- `LeaveOneOut`

- `LeavePOut` 

- `RepeatedKFold`

-  `RepeatedStratifiedKFold`

- `ShuffleSplit`

I suggest updating it to the following, as this is how it has been updated in other classes.

```python
class KFold(...):
    def split(self, X, y=None, groups=None):
        """...

        groups : object
            Always ignored, exists for compatibility.
        ...
        """
``` 

Also, do we want to update the docstring for both the `get_n_splits` and `split` and method?

Labels: Documentation, Moderate, module:model_selection
KFold splitter falsely claims it supports groups

@thomasjpfan : (taken from https://github.com/scikit-learn/scikit-learn/pull/22765/files#r825487547)

> My mind is a little blown. So the current API for KFold.split has a groups parameter it never uses. Because of the inheritance structure, the [API docs for KFold.split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html#sklearn.model_selection.KFold.split) says the group labels are used in KFold.
> 
> (This comment is not actionable)


I haven't looked at how to proceed here. It certainly shouldn't claim it supports `groups` in the docs, and we need to check how to fix the API if we can.

COMMENT:
Hi all, I recently came across this issue and wanted to bump it. I would recommend not only changing the docstring, but also throwing an error or at least a warning whenever a user provided anything other than `groups=None` for the non-Group-samplers (KFold, StratifiedKFold, ShuffleSplit, etc). 

I recommend the additional warning because even when the documentation is clear (like in StratifiedKFold) the split functions will still silently ignore user's input which might lead to unintended consequences. This seems like a good place to have `if groups is not None: warnings.warn("this is ignored; see documentation")` or even throw an error.

I can offer to attempt to push some code to resolve this issue, if you'd like. Would that be helpful?

Labels: Documentation, Moderate, module:model_selection
Bunch object appears as :func: role in intersphinx inventory

### Describe the bug

The `Bunch` object appears with the `:py:func:` role in the `objects.inv` file used by intersphinx to reference objects from other projects. As it is a class, I think that it should appear with the `:py:class:` role instead. Otherwise one cannot use `` :external:class:`sklearn.utils.Bunch` `` and needs to use `` :external:obj:`sklearn.utils.Bunch` `` instead to achieve the same result.



### Steps/Code to Reproduce

```rst
Referencing :external:class:`sklearn.utils.Bunch` in another project.
```

### Expected Results

A link to the `Bunch` documentation is generated.

### Actual Results

Intersphinx reports the following warning and no link is generated:
```
WARNING: external py:class reference target not found: sklearn.utils.Bunch
```

### Versions

```shell
System:
    python: 3.8.0 | packaged by conda-forge | (default, Nov 22 2019, 19:11:38)  [GCC 7.3.0]
executable: /home/carlos/Programas/Utilidades/Lenguajes/miniconda3/envs/fda38/bin/python3.8
   machine: Linux-5.4.0-92-generic-x86_64-with-glibc2.10

Python dependencies:
          pip: 21.3.1
   setuptools: 60.5.0
      sklearn: 1.0.2
        numpy: 1.22.0
        scipy: 1.7.3
       Cython: 0.29.26
       pandas: 1.4.0
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.0.0

Built with OpenMP: True
```

COMMENT:
Thank you for the report! I agree this is a bug and opened https://github.com/scikit-learn/scikit-learn/pull/22948 to fix it.

Labels: Bug
Formula for dual gap of elastic net in coordinate descent solver

### Describe the bug
The computation of the dual gap for the elastic net in the coordinate descent solver (`enet_coordinate_descent`) might be wrong.

The elastic net minimizes
```
Primal(w) = (1/2) * ||y - X w||_2^2 + alpha * ||w||_1 + beta/2 * ||w||_2^2
```

### Lasso
For the pure Lasso, i.e. `beta=0`, the dual becomes [1]
```
Dual(nu) = -1/2 * ||nu||_2^2 - nu'y   if ||X'nu||_∞ = max(abs(X'nu)) <= alpha   else -∞
```
with `nu = Xw - y` (=minus the residual `R`) as possible dual point. This yields the Lasso dual gap
```
Primal(w) - Dual(nu)
```
In case of `||X'nu||_∞ > alpha`, one uses a down-scaled variable `nu` in the dual.

### Elastic Net
For the elastic net, the dual, see section 5.2.3 in [2], becomes
```
Dual(nu) = -1/2 * ||nu||_2^2 - nu'y - 1/(2*beta) * sum_j (|X_j'nu| - alpha)_+^2
```
with `X_j` the j-th column of `X` and `x_+ = max(0, x)` the positive part.

### References
[1] Kim, Seung-Jean et al. “An Interior-Point Method for Large-Scale $\ell_1$-Regularized Least Squares.” IEEE Journal of Selected Topics in Signal Processing 1 (2007): 606-617. [pdf link](http://stanford.edu/~boyd/papers/pdf/l1_ls.pdf)
[2] Mendler-Dünner, Celestine et al. “Primal-Dual Rates and Certificates.” ICML (2016). [arxiv link](https://arxiv.org/pdf/1602.05205.pdf)

### Expected Results

```python
if beta > =0:
    R = y - X @ w  # note: R = -nu
    R_norm2 = R @ R
    gap = R_norm2 - R @ y + alpha * l1_norm + beta/2 * l2_norm
    gap += 1/(2*beta) * np.sum(np.max(0, np.abs(X.T @ R) - alpha)**2)
```
At least a test for it, similar to the pure Lasso case in https://github.com/scikit-learn/scikit-learn/blob/2c2f31d68c21b7647557b2f776e86c05954d80bf/sklearn/linear_model/tests/test_coordinate_descent.py#L292

### Actual Results

I start to be rattled by the `-beta * w` term in line 205.
https://github.com/scikit-learn/scikit-learn/blob/2c2f31d68c21b7647557b2f776e86c05954d80bf/sklearn/linear_model/_cd_fast.pyx#L205-L236

COMMENT:
@agramfort @mathurinm @TomDLT Your help would be highly appreciated to figure out the right dual gap for the elastic net. Above, I gave references and also did the computation myself. Pure Lasso seems ok. I can't make sense of the dual gap computation in scikit-learn as soon as we have a true elastic net (L1 + L2 penalty).

Labels: Bug, module:linear_model
Formula for dual gap of elastic net in coordinate descent solver

### Describe the bug
The computation of the dual gap for the elastic net in the coordinate descent solver (`enet_coordinate_descent`) might be wrong.

The elastic net minimizes
```
Primal(w) = (1/2) * ||y - X w||_2^2 + alpha * ||w||_1 + beta/2 * ||w||_2^2
```

### Lasso
For the pure Lasso, i.e. `beta=0`, the dual becomes [1]
```
Dual(nu) = -1/2 * ||nu||_2^2 - nu'y   if ||X'nu||_∞ = max(abs(X'nu)) <= alpha   else -∞
```
with `nu = Xw - y` (=minus the residual `R`) as possible dual point. This yields the Lasso dual gap
```
Primal(w) - Dual(nu)
```
In case of `||X'nu||_∞ > alpha`, one uses a down-scaled variable `nu` in the dual.

### Elastic Net
For the elastic net, the dual, see section 5.2.3 in [2], becomes
```
Dual(nu) = -1/2 * ||nu||_2^2 - nu'y - 1/(2*beta) * sum_j (|X_j'nu| - alpha)_+^2
```
with `X_j` the j-th column of `X` and `x_+ = max(0, x)` the positive part.

### References
[1] Kim, Seung-Jean et al. “An Interior-Point Method for Large-Scale $\ell_1$-Regularized Least Squares.” IEEE Journal of Selected Topics in Signal Processing 1 (2007): 606-617. [pdf link](http://stanford.edu/~boyd/papers/pdf/l1_ls.pdf)
[2] Mendler-Dünner, Celestine et al. “Primal-Dual Rates and Certificates.” ICML (2016). [arxiv link](https://arxiv.org/pdf/1602.05205.pdf)

### Expected Results

```python
if beta > =0:
    R = y - X @ w  # note: R = -nu
    R_norm2 = R @ R
    gap = R_norm2 - R @ y + alpha * l1_norm + beta/2 * l2_norm
    gap += 1/(2*beta) * np.sum(np.max(0, np.abs(X.T @ R) - alpha)**2)
```
At least a test for it, similar to the pure Lasso case in https://github.com/scikit-learn/scikit-learn/blob/2c2f31d68c21b7647557b2f776e86c05954d80bf/sklearn/linear_model/tests/test_coordinate_descent.py#L292

### Actual Results

I start to be rattled by the `-beta * w` term in line 205.
https://github.com/scikit-learn/scikit-learn/blob/2c2f31d68c21b7647557b2f776e86c05954d80bf/sklearn/linear_model/_cd_fast.pyx#L205-L236

COMMENT:
I think it's based on the trick where you write the Elastic net as a Lasso, with an extended design X (shape (n_samples + n_features, n_features). The Enet objective (without the n_samples dividing the datafit) is a Lasso with following design matrices and observation vector: 
```python
X_tilde = np.vstack([X, np.sqrt(beta) * np.eye(n_features)])
y_tilde = np.hstack([y, np.zeros(n_features)])
```


and so `X_tilde.T @ (X_tilde @ w - y_tilde) = X_tilde.T @ np.vstack([X @ w - y, np.sqrt(beta) * w]) = X.T @ (X @ w - y) + beta * w`

Labels: Bug, module:linear_model
Formula for dual gap of elastic net in coordinate descent solver

### Describe the bug
The computation of the dual gap for the elastic net in the coordinate descent solver (`enet_coordinate_descent`) might be wrong.

The elastic net minimizes
```
Primal(w) = (1/2) * ||y - X w||_2^2 + alpha * ||w||_1 + beta/2 * ||w||_2^2
```

### Lasso
For the pure Lasso, i.e. `beta=0`, the dual becomes [1]
```
Dual(nu) = -1/2 * ||nu||_2^2 - nu'y   if ||X'nu||_∞ = max(abs(X'nu)) <= alpha   else -∞
```
with `nu = Xw - y` (=minus the residual `R`) as possible dual point. This yields the Lasso dual gap
```
Primal(w) - Dual(nu)
```
In case of `||X'nu||_∞ > alpha`, one uses a down-scaled variable `nu` in the dual.

### Elastic Net
For the elastic net, the dual, see section 5.2.3 in [2], becomes
```
Dual(nu) = -1/2 * ||nu||_2^2 - nu'y - 1/(2*beta) * sum_j (|X_j'nu| - alpha)_+^2
```
with `X_j` the j-th column of `X` and `x_+ = max(0, x)` the positive part.

### References
[1] Kim, Seung-Jean et al. “An Interior-Point Method for Large-Scale $\ell_1$-Regularized Least Squares.” IEEE Journal of Selected Topics in Signal Processing 1 (2007): 606-617. [pdf link](http://stanford.edu/~boyd/papers/pdf/l1_ls.pdf)
[2] Mendler-Dünner, Celestine et al. “Primal-Dual Rates and Certificates.” ICML (2016). [arxiv link](https://arxiv.org/pdf/1602.05205.pdf)

### Expected Results

```python
if beta > =0:
    R = y - X @ w  # note: R = -nu
    R_norm2 = R @ R
    gap = R_norm2 - R @ y + alpha * l1_norm + beta/2 * l2_norm
    gap += 1/(2*beta) * np.sum(np.max(0, np.abs(X.T @ R) - alpha)**2)
```
At least a test for it, similar to the pure Lasso case in https://github.com/scikit-learn/scikit-learn/blob/2c2f31d68c21b7647557b2f776e86c05954d80bf/sklearn/linear_model/tests/test_coordinate_descent.py#L292

### Actual Results

I start to be rattled by the `-beta * w` term in line 205.
https://github.com/scikit-learn/scikit-learn/blob/2c2f31d68c21b7647557b2f776e86c05954d80bf/sklearn/linear_model/_cd_fast.pyx#L205-L236

COMMENT:
indeed !

> Message ID: ***@***.***>
>

Labels: Bug, module:linear_model
Formula for dual gap of elastic net in coordinate descent solver

### Describe the bug
The computation of the dual gap for the elastic net in the coordinate descent solver (`enet_coordinate_descent`) might be wrong.

The elastic net minimizes
```
Primal(w) = (1/2) * ||y - X w||_2^2 + alpha * ||w||_1 + beta/2 * ||w||_2^2
```

### Lasso
For the pure Lasso, i.e. `beta=0`, the dual becomes [1]
```
Dual(nu) = -1/2 * ||nu||_2^2 - nu'y   if ||X'nu||_∞ = max(abs(X'nu)) <= alpha   else -∞
```
with `nu = Xw - y` (=minus the residual `R`) as possible dual point. This yields the Lasso dual gap
```
Primal(w) - Dual(nu)
```
In case of `||X'nu||_∞ > alpha`, one uses a down-scaled variable `nu` in the dual.

### Elastic Net
For the elastic net, the dual, see section 5.2.3 in [2], becomes
```
Dual(nu) = -1/2 * ||nu||_2^2 - nu'y - 1/(2*beta) * sum_j (|X_j'nu| - alpha)_+^2
```
with `X_j` the j-th column of `X` and `x_+ = max(0, x)` the positive part.

### References
[1] Kim, Seung-Jean et al. “An Interior-Point Method for Large-Scale $\ell_1$-Regularized Least Squares.” IEEE Journal of Selected Topics in Signal Processing 1 (2007): 606-617. [pdf link](http://stanford.edu/~boyd/papers/pdf/l1_ls.pdf)
[2] Mendler-Dünner, Celestine et al. “Primal-Dual Rates and Certificates.” ICML (2016). [arxiv link](https://arxiv.org/pdf/1602.05205.pdf)

### Expected Results

```python
if beta > =0:
    R = y - X @ w  # note: R = -nu
    R_norm2 = R @ R
    gap = R_norm2 - R @ y + alpha * l1_norm + beta/2 * l2_norm
    gap += 1/(2*beta) * np.sum(np.max(0, np.abs(X.T @ R) - alpha)**2)
```
At least a test for it, similar to the pure Lasso case in https://github.com/scikit-learn/scikit-learn/blob/2c2f31d68c21b7647557b2f776e86c05954d80bf/sklearn/linear_model/tests/test_coordinate_descent.py#L292

### Actual Results

I start to be rattled by the `-beta * w` term in line 205.
https://github.com/scikit-learn/scikit-learn/blob/2c2f31d68c21b7647557b2f776e86c05954d80bf/sklearn/linear_model/_cd_fast.pyx#L205-L236

COMMENT:
@mathurinm Thanks for the prompt reply! That's a great trick that I would have never guessed (even though I used it myself in the quantile regression). I will open a PR to document it a little better.

I'm still trying to figure out why those two formulations give different dual Lagragians, i.e. the derivation above is always finite (feasible set equal real numbers) while the Lasso trick needs `||X'nu + beta * w||_∞ <= alpha`. I understand that both have a different dual variable, `nu` vs `nu_tilde`. Does one formulation give a tighter dual gap bound than the other?

Labels: Bug, module:linear_model