Talip7/scikit-learn-issues-multilabel huggingface dataset:
Model Persistence page is missing side navigation contents

### Describe the issue linked to the documentation

Under User Guide, when you click the Model Persistence page, the left parent navigation contents is missing. 
Link: https://scikit-learn.org/stable/modules/model_persistence.html

I noticed the Model Persistence page is under Modules, which is why the `:parenttoc: True` won't work.

![image](https://user-images.githubusercontent.com/39132687/153765496-7c8d4d1f-e382-4b6c-897e-a19367a8776d.png)


### Suggest a potential alternative/fix

- Specify the parent of parent or table of contents. Or,
- Move the Model Persistence page under User Guide instead of modules (might break other documentation) so I wouldn't recommend it

COMMENT:
This is an example of when you click a different user guide section. The parent contents remains
![image](https://user-images.githubusercontent.com/39132687/153765572-c8fac836-34a0-4878-917c-e58e70d3aa95.png)

Labels: Documentation
Model Persistence page is missing side navigation contents

### Describe the issue linked to the documentation

Under User Guide, when you click the Model Persistence page, the left parent navigation contents is missing. 
Link: https://scikit-learn.org/stable/modules/model_persistence.html

I noticed the Model Persistence page is under Modules, which is why the `:parenttoc: True` won't work.

![image](https://user-images.githubusercontent.com/39132687/153765496-7c8d4d1f-e382-4b6c-897e-a19367a8776d.png)


### Suggest a potential alternative/fix

- Specify the parent of parent or table of contents. Or,
- Move the Model Persistence page under User Guide instead of modules (might break other documentation) so I wouldn't recommend it

COMMENT:
@wafiqsyed  working on this issue

Labels: Documentation
Model Persistence page is missing side navigation contents

### Describe the issue linked to the documentation

Under User Guide, when you click the Model Persistence page, the left parent navigation contents is missing. 
Link: https://scikit-learn.org/stable/modules/model_persistence.html

I noticed the Model Persistence page is under Modules, which is why the `:parenttoc: True` won't work.

![image](https://user-images.githubusercontent.com/39132687/153765496-7c8d4d1f-e382-4b6c-897e-a19367a8776d.png)


### Suggest a potential alternative/fix

- Specify the parent of parent or table of contents. Or,
- Move the Model Persistence page under User Guide instead of modules (might break other documentation) so I wouldn't recommend it

COMMENT:
Sorry @soma2000-lang, it looks like we were looking into the issue around the same time.

I opened #22511 which moves the `model_persistence` page up so the TOC shows on the left. The PR also redirects from the old page to the new location, so old links still work.

Labels: Documentation
The weighted average should be replaced with a weighted sum?

https://github.com/scikit-learn/scikit-learn/blob/7e1e6d09bcc2eaeba98f7e737aac2ac782f0e5f1/sklearn/metrics/_regression.py#L454

COMMENT:
This was asked before in https://github.com/scikit-learn/scikit-learn/issues/8758. The comment there applies here too: https://github.com/scikit-learn/scikit-learn/issues/8758#issuecomment-294809889

> do you have a reference for that? Presumably the difference is merely a
multiplicative constant per dataset, so rankings don't change or anything,
but difference venues may have different conventions for reporting

For me, I am -1 for changing this average because it is backward incompatible and `mean_squared_error` is a widely used function. Also `mean_squared_error` does allow for `multioutput="raw_values"`, so one can create a function to sum if they needed to.

Labels: module:metrics
The weighted average should be replaced with a weighted sum?

https://github.com/scikit-learn/scikit-learn/blob/7e1e6d09bcc2eaeba98f7e737aac2ac782f0e5f1/sklearn/metrics/_regression.py#L454

COMMENT:
The minimum of the coded function is the same as the minimum of what is typically defined as mean square error, since, as stated, the difference is only a constant factor. However, it is confusing to give a method a name which has a well-defined meaning in many applications and then code something else.

Labels: module:metrics
The weighted average should be replaced with a weighted sum?

https://github.com/scikit-learn/scikit-learn/blob/7e1e6d09bcc2eaeba98f7e737aac2ac782f0e5f1/sklearn/metrics/_regression.py#L454

COMMENT:
I am -1 for this change as well. Note that for consistency, such a change would affect not only `mean_squared_error` but also most other scoring metrics in `sklearn/metrics/_regression.py`.

The difference is a constant factor that can be easily implemented by the user. The name of the function actually starts with "mean", so I find it more intuitive to do essentially `squared_error.mean(axis=0).mean(axis=0)` rather than `squared_error.mean(axis=0).sum(axis=0)`.

Labels: module:metrics
The weighted average should be replaced with a weighted sum?

https://github.com/scikit-learn/scikit-learn/blob/7e1e6d09bcc2eaeba98f7e737aac2ac782f0e5f1/sklearn/metrics/_regression.py#L454

COMMENT:
Given the comments in https://github.com/scikit-learn/scikit-learn/issues/8758#issuecomment-294809889 and https://github.com/scikit-learn/scikit-learn/issues/22466#issuecomment-1040861874, I am closing this issue.

Labels: module:metrics
add which solver was used in sklearn Ridge regression : add clf.get_params(solver)

### Describe the issue linked to the documentation

https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html

solver{‘auto’, ‘svd’, ‘cholesky’, ‘lsqr’, ‘sparse_cg’, ‘sag’, ‘saga’, ‘lbfgs’}, default=’auto’
Solver to use in the computational routines:

‘auto’ chooses the solver automatically based on the type of data.

### Suggest a potential alternative/fix

add how to find which solver actually was used

for example

```
>>> from sklearn.linear_model import Ridge
>>> import numpy as np
>>> n_samples, n_features = 10, 5
>>> rng = np.random.RandomState(0)
>>> y = rng.randn(n_samples)
>>> X = rng.randn(n_samples, n_features)
>>> clf = Ridge(alpha=1.0)
>>> clf.fit(X, y)
Ridge()
add
clf.get_params(solver)
```


Original Link: https://github.com/scikit-learn/scikit-learn/issues/22341

COMMENT:
Sorry, this was posted here by mistake.

Labels: Needs Triage
Verbosity for OneVsRestClassifier

### Describe the workflow you want to enable

Hi, is it possible to add a verbose parameter to OneVsRestClassifier so that we can see what the model is currently doing?


### Describe your proposed solution

Add a parameter like 
```
OneVsRestClassifier(XGBClassifier(n_jobs=-1, max_depth=4, verbosity=2), verbose=1)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I think it's reasonable to allow for `verbose` in `OneVsRestClassifier` and pass it in when constructing the `Parallel` object:

https://github.com/scikit-learn/scikit-learn/blob/6c09bab585acc708b099acf383fefe6ffe50ca89/sklearn/multiclass.py#L308

Labels: New Feature, module:multiclass
Verbosity for OneVsRestClassifier

### Describe the workflow you want to enable

Hi, is it possible to add a verbose parameter to OneVsRestClassifier so that we can see what the model is currently doing?


### Describe your proposed solution

Add a parameter like 
```
OneVsRestClassifier(XGBClassifier(n_jobs=-1, max_depth=4, verbosity=2), verbose=1)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@thomasjpfan I can work on implementing this. 

`joblib` has a somewhat strange interface for their verbosity, basically printing to stderr when `verbose < 50` and increasing the frequency of iterations printed until all iterations are printed when `verbose >= 10`. Should this interface be exposed directly, or should I just do a simple verbose where `verbose == 0` means no printing and `verbose != 0` means everything is printed? Seems a bit strange to break verbosity conventions that are held throughout the rest of the library, don't want someone passing in `verbose=1` and not realizing all iterations aren't being printed.

Labels: New Feature, module:multiclass
Verbosity for OneVsRestClassifier

### Describe the workflow you want to enable

Hi, is it possible to add a verbose parameter to OneVsRestClassifier so that we can see what the model is currently doing?


### Describe your proposed solution

Add a parameter like 
```
OneVsRestClassifier(XGBClassifier(n_jobs=-1, max_depth=4, verbosity=2), verbose=1)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
In general, we are not consistent with `verbose`, but there is some consistency with passing `verbose` to `Parallel`. If we are passing `verbose` directly to `Parallel`, then we normally do not do anything and pass it in directly. For example:

https://github.com/scikit-learn/scikit-learn/blob/e11c4d21a4579f0d49f414a4b76e386f80f0f074/sklearn/ensemble/_bagging.py#L394-L396

To help with some confusion, we can link `joblib.Parallel` in the docstring for `verbose` so users know about this behavior.

Labels: New Feature, module:multiclass
FEA post-fit calibration option in HGBT

### Describe the workflow you want to enable

The histogram gradient boosted decision trees usually do not fulfil the so called *balance property* on the training data, i.e. `sum([proba]predictions) == sum(observations)`. A simple "post-fit" step could ensure this condition. This will usually decrease the in-sample performance, but I observed in practise that is is quite beneficial for the out-of-sample performance, in particular for non-canonical link-loss combinations such as the Gamma deviance with log link (with XGBoost or LightGBM as it is not yet available in scikit-learn). The main advantage is, however, better calibrated models, in-sample as well as out-of-sample.
```python
model = HistGradientBoostingRegressor()  # could also be a classifier
model.fit(X, y)
# post-fit recalibration to fulfil the balance property
model._baseline_prediction = link_function(np.mean(y) / np.mean(model.predit[_proba](X))
```
For quantiles, this would be slightly different.

### Describe your proposed solution

Add a new option `post_fit_calibration` (better name?!):
```python
model = HistGradientBoostingRegressor(post_fit_calibration=True)  # could also be a classifier
model.fit(X, y)
```

### Describe alternatives you've considered, if relevant

One could also invent a meta-estimator.

### Additional context

_No response_

COMMENT:
If we had this option, we should really include a new example or expand an existing example to teach our users about both the theoretical motivation and the practical impacts of this non-trivial option.

Labels: New Feature, module:ensemble
FEA post-fit calibration option in HGBT

### Describe the workflow you want to enable

The histogram gradient boosted decision trees usually do not fulfil the so called *balance property* on the training data, i.e. `sum([proba]predictions) == sum(observations)`. A simple "post-fit" step could ensure this condition. This will usually decrease the in-sample performance, but I observed in practise that is is quite beneficial for the out-of-sample performance, in particular for non-canonical link-loss combinations such as the Gamma deviance with log link (with XGBoost or LightGBM as it is not yet available in scikit-learn). The main advantage is, however, better calibrated models, in-sample as well as out-of-sample.
```python
model = HistGradientBoostingRegressor()  # could also be a classifier
model.fit(X, y)
# post-fit recalibration to fulfil the balance property
model._baseline_prediction = link_function(np.mean(y) / np.mean(model.predit[_proba](X))
```
For quantiles, this would be slightly different.

### Describe your proposed solution

Add a new option `post_fit_calibration` (better name?!):
```python
model = HistGradientBoostingRegressor(post_fit_calibration=True)  # could also be a classifier
model.fit(X, y)
```

### Describe alternatives you've considered, if relevant

One could also invent a meta-estimator.

### Additional context

_No response_

COMMENT:
API-wise, should it be an option (in HGBT, but also GammaRegressor and TweedieRegressor), or a meta-estimator?

I'm happy with a parameter per estimator and one PR per estimator where it makes sense.

Labels: New Feature, module:ensemble
PCA hangs occasionally when applied to data that fits in memory

### Describe the bug

When trying to `fit` PCA with a large enough array that fits in memory (approximately `300 000 x 750`) with an aim to make about 3 times reduction, unstable results are obtained: in most cases the *fit succeeds*, but occasionaly it *hangs "forever"*.

Random seed is fixed, but unfortunalely it does not help in this case.
The `svd_solver` option is left at the default. Looks like explicitly specifying `svd_solver='arpack'` does not cause this issue.

### Steps/Code to Reproduce

```python
import numpy as np
import random

np.random.seed(42)
random.seed(42)

from sklearn.decomposition import PCA


if __name__ == '__main__':
    embeddings = np.random.RandomState(42).normal(size=(328039, 768))
    embeddings = embeddings.astype(np.float32)

    for i in range(20):
        print('Iteration', i)
        pca = PCA(n_components=260, random_state=42)
        pca = pca.fit(embeddings)

```

### Expected Results

Stable results: consistently successful fits over a finite time.

### Actual Results

Sometimes the process hangs, in most cases in the first 10 iterations. I haven't found any consistency pattern at what time it fails.

### Versions

```shell
System:
   python: 3.9.7 | packaged by conda-forge 
   machine: Windows-10-10.0.19043-SP0
   RAM: 32Gb

Python dependencies:
          pip: 21.3.1
   setuptools: 60.8.1
      sklearn: 1.0.1
        numpy: 1.20.1
        scipy: 1.7.1
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
I cannot reproduce with the following system and packages version:

```
System:
    python: 3.8.12 | packaged by conda-forge | (default, Sep 16 2021, 01:38:21)  [Clang 11.1.0 ]
executable: /Users/glemaitre/mambaforge/envs/dev/bin/python
   machine: macOS-12.1-arm64-arm-64bit

Python dependencies:
          pip: 21.3
   setuptools: 58.2.0
      sklearn: 1.1.dev0
        numpy: 1.21.2
        scipy: 1.8.0.dev0+1902.b795164
       Cython: 0.29.24
       pandas: 1.3.3
   matplotlib: 3.4.3
       joblib: 1.2.0.dev0
threadpoolctl: 3.0.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /Users/glemaitre/mambaforge/envs/dev/lib/libopenblas_vortexp-r0.3.18.dylib
        version: 0.3.18
threading_layer: openmp
   architecture: VORTEX
    num_threads: 8

       user_api: openmp
   internal_api: openmp
         prefix: libomp
       filepath: /Users/glemaitre/mambaforge/envs/dev/lib/libomp.dylib
        version: None
    num_threads: 8
```

Labels: Bug, module:decomposition, Needs Investigation
PCA hangs occasionally when applied to data that fits in memory

### Describe the bug

When trying to `fit` PCA with a large enough array that fits in memory (approximately `300 000 x 750`) with an aim to make about 3 times reduction, unstable results are obtained: in most cases the *fit succeeds*, but occasionaly it *hangs "forever"*.

Random seed is fixed, but unfortunalely it does not help in this case.
The `svd_solver` option is left at the default. Looks like explicitly specifying `svd_solver='arpack'` does not cause this issue.

### Steps/Code to Reproduce

```python
import numpy as np
import random

np.random.seed(42)
random.seed(42)

from sklearn.decomposition import PCA


if __name__ == '__main__':
    embeddings = np.random.RandomState(42).normal(size=(328039, 768))
    embeddings = embeddings.astype(np.float32)

    for i in range(20):
        print('Iteration', i)
        pca = PCA(n_components=260, random_state=42)
        pca = pca.fit(embeddings)

```

### Expected Results

Stable results: consistently successful fits over a finite time.

### Actual Results

Sometimes the process hangs, in most cases in the first 10 iterations. I haven't found any consistency pattern at what time it fails.

### Versions

```shell
System:
   python: 3.9.7 | packaged by conda-forge 
   machine: Windows-10-10.0.19043-SP0
   RAM: 32Gb

Python dependencies:
          pip: 21.3.1
   setuptools: 60.8.1
      sklearn: 1.0.1
        numpy: 1.20.1
        scipy: 1.7.1
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
Here is a workaround

```
import numpy as np
from multiprocessing import Pool, TimeoutError
from sklearn.decomposition import PCA

rng = np.random.default_rng(42)
X = rng.normal(size=(328039, 768)).astype(np.float32)

def fit_pca(X):
    return PCA(n_components=260, random_state=42).fit(X)

for i in range(20):
    print(f'Iteration {i}')
    with Pool(processes=1) as pool:
        res = pool.apply_async(fit_pca, (X,))
        try:
            pca = res.get(timeout=10)
        except TimeoutError:
            print('Timed out')
            pca = None
    print(pca)
```

Interestingly, I've found that when PCA is fit many time in the same loop with fewer computation and disk I/O operations in between PCA fits, I run into issues but when I do a similar number of PCA fits with many more computations and I/O in between it consistently does not hang. Not sure if this means the issue is in some C/Cython code or what, I haven't looked but thought it was interesting and might help troublehshooting.

Labels: Bug, module:decomposition, Needs Investigation
PCA hangs occasionally when applied to data that fits in memory

### Describe the bug

When trying to `fit` PCA with a large enough array that fits in memory (approximately `300 000 x 750`) with an aim to make about 3 times reduction, unstable results are obtained: in most cases the *fit succeeds*, but occasionaly it *hangs "forever"*.

Random seed is fixed, but unfortunalely it does not help in this case.
The `svd_solver` option is left at the default. Looks like explicitly specifying `svd_solver='arpack'` does not cause this issue.

### Steps/Code to Reproduce

```python
import numpy as np
import random

np.random.seed(42)
random.seed(42)

from sklearn.decomposition import PCA


if __name__ == '__main__':
    embeddings = np.random.RandomState(42).normal(size=(328039, 768))
    embeddings = embeddings.astype(np.float32)

    for i in range(20):
        print('Iteration', i)
        pca = PCA(n_components=260, random_state=42)
        pca = pca.fit(embeddings)

```

### Expected Results

Stable results: consistently successful fits over a finite time.

### Actual Results

Sometimes the process hangs, in most cases in the first 10 iterations. I haven't found any consistency pattern at what time it fails.

### Versions

```shell
System:
   python: 3.9.7 | packaged by conda-forge 
   machine: Windows-10-10.0.19043-SP0
   RAM: 32Gb

Python dependencies:
          pip: 21.3.1
   setuptools: 60.8.1
      sklearn: 1.0.1
        numpy: 1.20.1
        scipy: 1.7.1
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
I discovered this issue myself and stumbled upon this issue thread. Seeing mentions of memory usage in this thread, I tried this approach:

```python
for i in range(loop):
    from sklearn.decomposition import PCA
    data = ...
    output = PCA(...).fit(data)
    del PCA
    del data
```
 Delete the data and PCA at the end of an iteration, and the code runs normally!

Does this mean there's a memory leak?

Labels: Bug, module:decomposition, Needs Investigation
PCA hangs occasionally when applied to data that fits in memory

### Describe the bug

When trying to `fit` PCA with a large enough array that fits in memory (approximately `300 000 x 750`) with an aim to make about 3 times reduction, unstable results are obtained: in most cases the *fit succeeds*, but occasionaly it *hangs "forever"*.

Random seed is fixed, but unfortunalely it does not help in this case.
The `svd_solver` option is left at the default. Looks like explicitly specifying `svd_solver='arpack'` does not cause this issue.

### Steps/Code to Reproduce

```python
import numpy as np
import random

np.random.seed(42)
random.seed(42)

from sklearn.decomposition import PCA


if __name__ == '__main__':
    embeddings = np.random.RandomState(42).normal(size=(328039, 768))
    embeddings = embeddings.astype(np.float32)

    for i in range(20):
        print('Iteration', i)
        pca = PCA(n_components=260, random_state=42)
        pca = pca.fit(embeddings)

```

### Expected Results

Stable results: consistently successful fits over a finite time.

### Actual Results

Sometimes the process hangs, in most cases in the first 10 iterations. I haven't found any consistency pattern at what time it fails.

### Versions

```shell
System:
   python: 3.9.7 | packaged by conda-forge 
   machine: Windows-10-10.0.19043-SP0
   RAM: 32Gb

Python dependencies:
          pip: 21.3.1
   setuptools: 60.8.1
      sklearn: 1.0.1
        numpy: 1.20.1
        scipy: 1.7.1
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
I can confirm that this is an issue with the threading backend - the sample provided runs normally if the backend number of threads is [forced to one using environment variables](https://scikit-learn.org/stable/computing/parallelism.html#parallel-numpy-and-scipy-routines-from-numerical-libraries). 

The workaround is using threadpoolctl to force single threaded mode:

```python
import numpy as np
import random
from threadpoolctl import threadpool_limits

np.random.seed(42)
random.seed(42)

from sklearn.decomposition import PCA


if __name__ == '__main__':
    embeddings = np.random.RandomState(42).normal(size=(328039, 768))
    embeddings = embeddings.astype(np.float32)

    for i in range(20):
        print('Iteration', i)

        with threadpool_limits(limits=1):
            pca = PCA(n_components=260, random_state=42)
            pca = pca.fit(embeddings)
```
but I'm not sure if this solution will work if there is also multithreaded code somewhere else in the script.

Labels: Bug, module:decomposition, Needs Investigation