Talip7/scikit-learn-issues-multilabel huggingface dataset:
clarification on OAS estimator formula (sklearn.covariance.OAS) (mean instead of trace is used)

Dear sklearn experts,

I was comparing different shrinkage algorithms and when looking at sklearn implementation of the OAS estimator I found something strange in the definition of the shrinkage factor or at least not clear to me. In the original formula from Chen et al. 2010 (the formula is also wrong in the paper, anyway) they used always the trace of the covariance matrix. Instead, this is the formula from sklearn.covariance.OAS module:

     mu = np.trace(emp_cov) / n_features
    # formula from Chen et al.'s **implementation**
    alpha = np.mean(emp_cov ** 2)
    num = alpha + mu ** 2
    den = (n_samples + 1.) * (alpha - (mu ** 2) / n_features)

    shrinkage = 1. if den == 0 else min(num / den, 1.)
    shrunk_cov = (1. - shrinkage) * emp_cov
    shrunk_cov.flat[::n_features + 1] += shrinkage * mu

where alpha is the mean of the squared covariance matrix instead of the trace, and also the mu parameter is normalized by the number of features also at the numerator, differently from what I found in the literature, referring to the same formula.
A trascription of what I found in papers should be (discarding the factor 2/p as in sklearn):

     mu = np.trace(emp_cov) 
    alpha = np.trace(emp_cov ** 2)
    num = alpha + mu ** 2
    den = (n_samples + 1.) * (alpha - (mu ** 2) / n_features)

    shrinkage = 1. if den == 0 else min(num / den, 1.)
    shrunk_cov = (1. - shrinkage) * emp_cov + shrinkage * np.diag(np.diag(emp_cov))
    

Is this right? Are the two forms equivalent in a way that I couldn't understand?

Thank you in advance,
Assunta

COMMENT:
Hey there @assuntaciarlo! Sorry for the long delay with this issue. After reading into [original publication](https://ieeexplore.ieee.org/document/8330356) (I believe it's been corrected) and considering the [R implementation](https://rdrr.io/cran/CovTools/src/R/CovEst.2010OAS.R) I'm inclined to agree with you that the current implementation is off. In particular, the  $\mu^2$ terms in the numerator and denominator resolve to  $\operatorname{tr} (\hat S)^2 / M^2$. Should be a fairly quick fix. Opened a PR here: #23867

Labels: module:covariance, Needs Investigation
partial_fit from SelectFromModel doesn't validate the parameters

### Describe the bug

Bug discovered while reviewing #23271.
in `SelectFromModel`, the `partial_fit` method doesn't do any validation. It should do the same validation as the `fit`method. It should also set ``n_features_in_`` and co.

### Steps/Code to Reproduce

```py
from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import SGDClassifier
X = [[ 0.87, -1.34,  0.31 ],
     [-2.79, -0.02, -0.85 ],
     [-1.34, -0.48, -2.55 ],
     [ 1.92,  1.48,  0.65 ]]
y = [0, 1, 0, 1]
selector = SelectFromModel(
    estimator=SGDClassifier(),
    max_features=10  # bad number of features
).partial_fit(X, y, classes=[0,1])
selector.transform(X)
```


### Expected Results

Error with informative error message, like `max_features should be less or equal than n_features`

### Actual Results

AttributeError: 'SelectFromModel' object has no attribute 'max_features_'

### Versions

```shell
latest
```

COMMENT:
cc/ @glemaitre

Labels: Bug, module:feature_selection
Sklearn python 3.7 failing on pip

### Describe the bug

Installation failed with python 3.7. found it on travis. Python other versions till 3.9 are working fine.

### Steps/Code to Reproduce

pip install sklearn

### Expected Results

Successful installtion

### Actual Results

Collecting sklearn

  Downloading sklearn-0.0.tar.gz (1.1 kB)

  Preparing metadata (setup.py): started

  Preparing metadata (setup.py): finished with status 'error'

  error: subprocess-exited-with-error

  

  × [secure] setup.py egg_info did not run successfully.

  │ exit code: 1

  ╰─> [22 lines of output]

      Traceback (most recent call last):

        File "<string>", line 36, in <module>

        File "<pip-setuptools-caller>", line 14, in <module>

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/setuptools/__init__.py", line 18, in <module>

          from setuptools.dist import Distribution

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/setuptools/dist.py", line 34, in <module>

          from ._importlib import metadata

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/setuptools/_importlib.py", line 28, in <module>

          disable_importlib_metadata_finder(metadata)

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/setuptools/_importlib.py", line 12, in disable_importlib_metadata_finder

          import importlib_metadata

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/importlib_metadata/__init__.py", line 471, in <module>

          __version__ = version(__name__)

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/importlib_metadata/__init__.py", line 438, in version

          return distribution(package).version

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/importlib_metadata/__init__.py", line 411, in distribution

          return Distribution.from_name(package)

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/importlib_metadata/__init__.py", line 179, in from_name

          dists = resolver(name)

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 886, in find_distributions

          found = self._search_paths(context.name, context.path)

      AttributeError: 'str' object has no attribute 'name'

      [end of output]

  

  note: This error originates from a subprocess, and is likely not a problem with pip.

error: metadata-generation-failed

× Encountered error while generating package metadata.

╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.

hint: See above for details.

The command "pip install sklearn" failed and exited with 1 during .

### Versions

```shell
sklearn for python 3.7
```

COMMENT:
Hi @Pranavkhade, the correct command to install scikit-learn is ``pip install scikit-learn``

Labels: Bug, Needs Triage
Sklearn python 3.7 failing on pip

### Describe the bug

Installation failed with python 3.7. found it on travis. Python other versions till 3.9 are working fine.

### Steps/Code to Reproduce

pip install sklearn

### Expected Results

Successful installtion

### Actual Results

Collecting sklearn

  Downloading sklearn-0.0.tar.gz (1.1 kB)

  Preparing metadata (setup.py): started

  Preparing metadata (setup.py): finished with status 'error'

  error: subprocess-exited-with-error

  

  × [secure] setup.py egg_info did not run successfully.

  │ exit code: 1

  ╰─> [22 lines of output]

      Traceback (most recent call last):

        File "<string>", line 36, in <module>

        File "<pip-setuptools-caller>", line 14, in <module>

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/setuptools/__init__.py", line 18, in <module>

          from setuptools.dist import Distribution

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/setuptools/dist.py", line 34, in <module>

          from ._importlib import metadata

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/setuptools/_importlib.py", line 28, in <module>

          disable_importlib_metadata_finder(metadata)

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/setuptools/_importlib.py", line 12, in disable_importlib_metadata_finder

          import importlib_metadata

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/importlib_metadata/__init__.py", line 471, in <module>

          __version__ = version(__name__)

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/importlib_metadata/__init__.py", line 438, in version

          return distribution(package).version

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/importlib_metadata/__init__.py", line 411, in distribution

          return Distribution.from_name(package)

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/importlib_metadata/__init__.py", line 179, in from_name

          dists = resolver(name)

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 886, in find_distributions

          found = self._search_paths(context.name, context.path)

      AttributeError: 'str' object has no attribute 'name'

      [end of output]

  

  note: This error originates from a subprocess, and is likely not a problem with pip.

error: metadata-generation-failed

× Encountered error while generating package metadata.

╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.

hint: See above for details.

The command "pip install sklearn" failed and exited with 1 during .

### Versions

```shell
sklearn for python 3.7
```

COMMENT:
I am adding the link to the install page here: https://scikit-learn.org/stable/install.html
and closing the issue.

Labels: Bug, Needs Triage
Sklearn python 3.7 failing on pip

### Describe the bug

Installation failed with python 3.7. found it on travis. Python other versions till 3.9 are working fine.

### Steps/Code to Reproduce

pip install sklearn

### Expected Results

Successful installtion

### Actual Results

Collecting sklearn

  Downloading sklearn-0.0.tar.gz (1.1 kB)

  Preparing metadata (setup.py): started

  Preparing metadata (setup.py): finished with status 'error'

  error: subprocess-exited-with-error

  

  × [secure] setup.py egg_info did not run successfully.

  │ exit code: 1

  ╰─> [22 lines of output]

      Traceback (most recent call last):

        File "<string>", line 36, in <module>

        File "<pip-setuptools-caller>", line 14, in <module>

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/setuptools/__init__.py", line 18, in <module>

          from setuptools.dist import Distribution

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/setuptools/dist.py", line 34, in <module>

          from ._importlib import metadata

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/setuptools/_importlib.py", line 28, in <module>

          disable_importlib_metadata_finder(metadata)

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/setuptools/_importlib.py", line 12, in disable_importlib_metadata_finder

          import importlib_metadata

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/importlib_metadata/__init__.py", line 471, in <module>

          __version__ = version(__name__)

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/importlib_metadata/__init__.py", line 438, in version

          return distribution(package).version

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/importlib_metadata/__init__.py", line 411, in distribution

          return Distribution.from_name(package)

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/importlib_metadata/__init__.py", line 179, in from_name

          dists = resolver(name)

        File "/home/travis/virtualenv/[secure]3.7.1/lib/[secure]3.7/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 886, in find_distributions

          found = self._search_paths(context.name, context.path)

      AttributeError: 'str' object has no attribute 'name'

      [end of output]

  

  note: This error originates from a subprocess, and is likely not a problem with pip.

error: metadata-generation-failed

× Encountered error while generating package metadata.

╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.

hint: See above for details.

The command "pip install sklearn" failed and exited with 1 during .

### Versions

```shell
sklearn for python 3.7
```

COMMENT:
Sorry about that guys.

Labels: Bug, Needs Triage
Add standard-deviation output to sklearn.ensemble.RandomForestRegressor

### Describe the workflow you want to enable

I think it would be awesome if the RF regressor returns the standard deviation (not only the mean) of the output of the different trees.


### Describe your proposed solution


This is not a definite option, but contains the core idea:

 

    from sklearn.ensemble import RandomForestRegressor
    import numpy as np
  
    class RandomForestWithUncertainty:

    def __init__(self,**args):

        self.model = RandomForestRegressor(**args)
    
    def predict(self, X, return_std = False):

        pred = np.array([tree.predict(X) for tree in self.model]).T
        pred_mean = np.mean(pred, axis=1)

        if return_std:

          pred_var = (pred - pred_mean.reshape(-1,1))**2
          pred_std = np.sqrt(np.mean(pred_var, axis=1))
          return pred_mean, pred_std

        else:
          return pred_mean





### Describe alternatives you've considered, if relevant

_No response_

### Additional context

This is useful if we want to use the RF in algorithms that require uncertainties estimatios, such as Bayesian Optimization.

COMMENT:
Personally, I think adding an option to return the deviation for bagging estimators, or even just returning the prediction from all the base estimators (without taking the average) so users can calculate the deviation themselves will be quite useful.  Mainly because the deviation of bagging estimators is indeed a common approach to model **epistemic uncertainty**.

Meanwhile using DecisionTree as base estimator is usually **NOT** a good choice for estimating epistemic uncertainty, due to the fact that tree model tends to predict similar values in a region where very few data is present (because no data can be used for splitting), as shown in the following image:

> Left hand side is a synthetic dataset (sin wave), right hand side is the output of model prediction along with estimated standard deviation from a bag of base estimators

- Tree model as base estimator
![image](https://user-images.githubusercontent.com/16646940/166433989-1a8a6433-6ba8-4d94-a49e-4c61188e22ea.png)

- NN as base estimator
![image](https://user-images.githubusercontent.com/16646940/166434537-24ead8af-85c0-458d-a961-bfab5f07efb8.png)

As shown in the result, tree model as base estimator performs poorly in estimating uncertainty.

Labels: New Feature, Needs Triage
Add standard-deviation output to sklearn.ensemble.RandomForestRegressor

### Describe the workflow you want to enable

I think it would be awesome if the RF regressor returns the standard deviation (not only the mean) of the output of the different trees.


### Describe your proposed solution


This is not a definite option, but contains the core idea:

 

    from sklearn.ensemble import RandomForestRegressor
    import numpy as np
  
    class RandomForestWithUncertainty:

    def __init__(self,**args):

        self.model = RandomForestRegressor(**args)
    
    def predict(self, X, return_std = False):

        pred = np.array([tree.predict(X) for tree in self.model]).T
        pred_mean = np.mean(pred, axis=1)

        if return_std:

          pred_var = (pred - pred_mean.reshape(-1,1))**2
          pred_std = np.sqrt(np.mean(pred_var, axis=1))
          return pred_mean, pred_std

        else:
          return pred_mean





### Describe alternatives you've considered, if relevant

_No response_

### Additional context

This is useful if we want to use the RF in algorithms that require uncertainties estimatios, such as Bayesian Optimization.

COMMENT:
Thank you for opening the issue. According to our [triaging guidelines](https://scikit-learn.org/dev/developers/bug_triaging.html#triaging-operations-for-members-of-the-core-and-contributor-experience-teams), I am closing this issue because it is a duplicate of https://github.com/scikit-learn/scikit-learn/issues/3271 which has a PR here: https://github.com/scikit-learn/scikit-learn/pull/5532.

Labels: New Feature, Needs Triage
RandomizedSearchCV verbose parameter description is not describing the verbosity levels.

### Describe the issue linked to the documentation

In the website of the RandomizedSearchCV the `verbose` parameter is not discussing the verbosity levels:
"verbose : int
Controls the verbosity: the higher, the more messages."
https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html

### Suggest a potential alternative/fix

We could imporove this as in the case of the GridSearchCV it is more detailed:
"verbose : int
Controls the verbosity: the higher, the more messages.
1 : the computation time for each fold and parameter candidate is displayed;
2 : the score is also displayed;
3 : the fold and candidate parameter indexes are also displayed together with the starting time of the computation."
https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html

COMMENT:
Thanks for the report @simonandras. Would you like to submit a PR to improve the doc ?

Labels: Documentation
RandomizedSearchCV verbose parameter description is not describing the verbosity levels.

### Describe the issue linked to the documentation

In the website of the RandomizedSearchCV the `verbose` parameter is not discussing the verbosity levels:
"verbose : int
Controls the verbosity: the higher, the more messages."
https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html

### Suggest a potential alternative/fix

We could imporove this as in the case of the GridSearchCV it is more detailed:
"verbose : int
Controls the verbosity: the higher, the more messages.
1 : the computation time for each fold and parameter candidate is displayed;
2 : the score is also displayed;
3 : the fold and candidate parameter indexes are also displayed together with the starting time of the computation."
https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html

COMMENT:
@jeremiedbb sure, I will do it.

Labels: Documentation
RandomizedSearchCV verbose parameter description is not describing the verbosity levels.

### Describe the issue linked to the documentation

In the website of the RandomizedSearchCV the `verbose` parameter is not discussing the verbosity levels:
"verbose : int
Controls the verbosity: the higher, the more messages."
https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html

### Suggest a potential alternative/fix

We could imporove this as in the case of the GridSearchCV it is more detailed:
"verbose : int
Controls the verbosity: the higher, the more messages.
1 : the computation time for each fold and parameter candidate is displayed;
2 : the score is also displayed;
3 : the fold and candidate parameter indexes are also displayed together with the starting time of the computation."
https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html

COMMENT:
Working on this

Labels: Documentation
Global config for n_jobs

### Describe the workflow you want to enable

As the title states, global config for n_jobs.

### Describe your proposed solution

I'd like a global config for n_jobs, where instead of me searching the documentation for every single method, I can maybe supply `N_JOBS=x` and that value gets passed on to every method that can support parallelization.

### Describe alternatives you've considered, if relevant

Manually adding `n_jobs=x` to every classifier/ regressor.

### Additional context

I work on a machine that has 128 cores. Sometimes I try to run a StackClassifier on 3 RandomForests only to realize that the RandomForestClassifer supports joblib parallelization, which could have worked wonders for my use case, but I couldn't.

COMMENT:
I am not entirely convinced that it would be a good idea. I can foresee 2 issues:

- oversubscription in case of nested parallelism even though `joblib` alleviates the issue.
- different estimators will scale differently in regard to the number of cores and it would be always better to set in a meaningful manner `n_jobs`.

@ogrisel might have more insights regarding this topic.

Labels: New Feature, Needs Triage
Global config for n_jobs

### Describe the workflow you want to enable

As the title states, global config for n_jobs.

### Describe your proposed solution

I'd like a global config for n_jobs, where instead of me searching the documentation for every single method, I can maybe supply `N_JOBS=x` and that value gets passed on to every method that can support parallelization.

### Describe alternatives you've considered, if relevant

Manually adding `n_jobs=x` to every classifier/ regressor.

### Additional context

I work on a machine that has 128 cores. Sometimes I try to run a StackClassifier on 3 RandomForests only to realize that the RandomForestClassifer supports joblib parallelization, which could have worked wonders for my use case, but I couldn't.

COMMENT:
I agree. Furthermore it's already possible to change the active backend and number of jobs globally with:

```python
from joblib import parallel_backend

parallel_backend("loky", n_jobs=4)

# do scikit-learn stuff here
```

or:

```python
from joblib import parallel_backend

parallel_backend("threading", n_jobs=4)

# do scikit-learn stuff here
```

Note that `parallel_backend` can either be called this way or a context manager.

Note that the threading backend is more efficient for random forests.

Labels: New Feature, Needs Triage
Global config for n_jobs

### Describe the workflow you want to enable

As the title states, global config for n_jobs.

### Describe your proposed solution

I'd like a global config for n_jobs, where instead of me searching the documentation for every single method, I can maybe supply `N_JOBS=x` and that value gets passed on to every method that can support parallelization.

### Describe alternatives you've considered, if relevant

Manually adding `n_jobs=x` to every classifier/ regressor.

### Additional context

I work on a machine that has 128 cores. Sometimes I try to run a StackClassifier on 3 RandomForests only to realize that the RandomForestClassifer supports joblib parallelization, which could have worked wonders for my use case, but I couldn't.

COMMENT:
Thanks for the help everyone!

Labels: New Feature, Needs Triage