Talip7/scikit-learn-issues-multilabel huggingface dataset:
Adding documentation about related library: Concrete ML, for privacy preserving ML

### Describe the issue linked to the documentation

We would like to add Concrete ML in the documentation. Concrete ML is an open-source package, providing privacy-preserving ML, thanks to so-called fully homomorphic encryption. Concrete ML contains so-called built-in models which are very very close to scikit-learn, XGB and skorch.  Eg, see https://docs.zama.ai/concrete-ml/built-in-models/linear 

We've started to discuss with some of you, who advised me to go for an issue. 

I would propose: 
- [x] adding a category Privacy Preserving Machine Learning in ./doc/related_projects.rst, and adding Concrete ML inside --- done, this was https://github.com/scikit-learn/scikit-learn/pull/27376
- [x] adding a category Privacy Preserving Machine Learning in examples, and copy some of the examples of https://github.com/zama-ai/concrete-ml/tree/main/docs/advanced_examples, or make new ones (what you prefer!) [not to be done]
- [ ] maybe propose a blog post, if you guest us. Will be done soon, https://github.com/zama-ai/concrete-ml-internal/issues/4033

and we are actually open to lot of suggestions.

What do you think?

### Suggest a potential alternative/fix

Adding some content as proposed in the description.

COMMENT:
Absolutely! It's https://github.com/zama-ai/concrete-ml-internal/issues/4033 in our internal repo. Stay tuned

Labels: Documentation
allow uniform intialization for BayesianGaussianMixture with dirichlet_process wight prior

### Describe the workflow you want to enable

I would like to have a deterministic version of the Bayesian Gaussian Mixture Model

### Describe your proposed solution

I propose allowing for unifrom responsibility initialization. This is possible when using the dirichlet process weight prior, because it is inherently asymmetric with the only assuption that there is an order to the cluster sizes

### Describe alternatives you've considered, if relevant

random initialization is non-deterministic which tends to introduce unwanted noise in the process

### Additional context

I have manually hacked in uniform initialzation, which is rather simple. it should likely be forbidden if the prior is not a dirichlet process but a symmetric dirichlet distribution

COMMENT:
I am not sure what this would entail, please feel free to open a PR and we will see how big a change this is.

It would make it helpful for the review to give a concrete application of this extension of the current constructor parameters.

Labels: New Feature
allow uniform intialization for BayesianGaussianMixture with dirichlet_process wight prior

### Describe the workflow you want to enable

I would like to have a deterministic version of the Bayesian Gaussian Mixture Model

### Describe your proposed solution

I propose allowing for unifrom responsibility initialization. This is possible when using the dirichlet process weight prior, because it is inherently asymmetric with the only assuption that there is an order to the cluster sizes

### Describe alternatives you've considered, if relevant

random initialization is non-deterministic which tends to introduce unwanted noise in the process

### Additional context

I have manually hacked in uniform initialzation, which is rather simple. it should likely be forbidden if the prior is not a dirichlet process but a symmetric dirichlet distribution

COMMENT:
It is actually a very small change. I will see if i can make an illustrative example. Might be a little while though.

Labels: New Feature
Redundant DataConversionWarning in BaseForest

### Describe the bug

A column vector is expected in `*Forest` even though the `y` is reshaped to be 2D:

```
        y = np.atleast_1d(y)
        if y.ndim == 2 and y.shape[1] == 1:
            warn(
                (
                    "A column-vector y was passed when a 1d array was"
                    " expected. Please change the shape of y to "
                    "(n_samples,), for example using ravel()."
                ),
                DataConversionWarning,
                stacklevel=2,
            )

        if y.ndim == 1:
            # reshape is necessary to preserve the data contiguity against vs
            # [:, np.newaxis] that does not.
            y = np.reshape(y, (-1, 1))
```

ref: https://github.com/scikit-learn/scikit-learn/blob/1d51c2eda47ba8db85f3d287ce2911986fe73d2b/sklearn/ensemble/_forest.py#L390-L405

### Steps/Code to Reproduce

```
from sklearn.ensemble import RandomForestClassifier

est = RandomForestClassifier()
est.fit(np.array([[0., 0.1], [1., 1.5]], y=np.array([0, 1]).reshape((-1, 1)))
```



### Expected Results

The warning seems redundant considering the array is transformed into a 2D array right afterwards anyways. I'm wondering if there is any reason to keep this warning here?

### Actual Results

```
E           sklearn.exceptions.DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().

sktree/_lib/sklearn/ensemble/_forest.py:467: DataConversionWarning
```

### Versions

```shell
>>> import sklearn; sklearn.show_versions()

System:
    python: 3.9.15 | packaged by conda-forge | (main, Nov 22 2022, 08:48:25)  [Clang 14.0.6 ]
executable: /Users/adam2392/miniforge3/envs/sktree/bin/python
   machine: macOS-13.5.2-arm64-arm-64bit

Python dependencies:
      sklearn: 1.3.0
          pip: 23.2.1
   setuptools: 65.7.0
        numpy: 1.25.2
        scipy: 1.10.1
       Cython: 0.29.36
       pandas: 2.0.3
   matplotlib: 3.7.2
       joblib: 1.3.2
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libomp
       filepath: /Users/adam2392/miniforge3/envs/sktree/lib/python3.9/site-packages/sklearn/.dylibs/libomp.dylib
        version: None
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /Users/adam2392/miniforge3/envs/sktree/lib/python3.9/site-packages/numpy/.dylibs/libopenblas64_.0.dylib
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: armv8
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /Users/adam2392/miniforge3/envs/sktree/lib/python3.9/site-packages/scipy/.dylibs/libopenblas.0.dylib
        version: 0.3.18
threading_layer: pthreads
   architecture: armv8
    num_threads: 8
```
```

COMMENT:
Ahh I see. Okay that makes sense then. I can close this issue.

Labels: Bug, Needs Triage
`make_sparse_spd_matrix` does not return sparse matrix

### Describe the bug

I have been looking at https://github.com/scikit-learn/scikit-learn/issues/27090 and working on `test_graphical_lasso.py`. There is used `make_sparse_spd_matrix` function, which, according to documentation, should return sparse matrix. But it returns `numpy.ndarray`. 

Did not find any issue mentioned it, so I considered it to be a bug. 
Should it return scipy sparse matrix, or documentation should be changed?

### Steps/Code to Reproduce

```python
from sklearn.datasets import make_sparse_spd_matrix
make_sparse_spd_matrix(4)
```

### Expected Results

scipy sparse matrix

### Actual Results

numpy.ndarray

### Versions

```shell
System:
    python: 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:42:20)  [Clang 14.0.6 ]
executable: /Users/y.korobko/mambaforge/envs/sklearn-env/bin/python
   machine: macOS-13.4.1-x86_64-i386-64bit
Python dependencies:
      sklearn: 1.4.dev0
          pip: 23.2.1
   setuptools: 68.0.0
        numpy: 1.25.1
        scipy: 1.11.1
       Cython: 3.0.0
       pandas: 2.0.3
   matplotlib: 3.7.2
       joblib: 1.3.1
threadpoolctl: 3.2.0
Built with OpenMP: False
threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 12
         prefix: libopenblas
       filepath: /Users/y.korobko/mambaforge/envs/sklearn-env/lib/libopenblasp-r0.3.23.dylib
        version: 0.3.23
threading_layer: openmp
   architecture: Haswell
       user_api: openmp
   internal_api: openmp
    num_threads: 12
         prefix: libomp
       filepath: /Users/y.korobko/mambaforge/envs/sklearn-env/lib/libomp.dylib
        version: None
```

COMMENT:
#27360 mentioned the same issue but with some solutions

Labels: Documentation
`make_sparse_spd_matrix` does not return sparse matrix

### Describe the bug

I have been looking at https://github.com/scikit-learn/scikit-learn/issues/27090 and working on `test_graphical_lasso.py`. There is used `make_sparse_spd_matrix` function, which, according to documentation, should return sparse matrix. But it returns `numpy.ndarray`. 

Did not find any issue mentioned it, so I considered it to be a bug. 
Should it return scipy sparse matrix, or documentation should be changed?

### Steps/Code to Reproduce

```python
from sklearn.datasets import make_sparse_spd_matrix
make_sparse_spd_matrix(4)
```

### Expected Results

scipy sparse matrix

### Actual Results

numpy.ndarray

### Versions

```shell
System:
    python: 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:42:20)  [Clang 14.0.6 ]
executable: /Users/y.korobko/mambaforge/envs/sklearn-env/bin/python
   machine: macOS-13.4.1-x86_64-i386-64bit
Python dependencies:
      sklearn: 1.4.dev0
          pip: 23.2.1
   setuptools: 68.0.0
        numpy: 1.25.1
        scipy: 1.11.1
       Cython: 3.0.0
       pandas: 2.0.3
   matplotlib: 3.7.2
       joblib: 1.3.1
threadpoolctl: 3.2.0
Built with OpenMP: False
threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 12
         prefix: libopenblas
       filepath: /Users/y.korobko/mambaforge/envs/sklearn-env/lib/libopenblasp-r0.3.23.dylib
        version: 0.3.23
threading_layer: openmp
   architecture: Haswell
       user_api: openmp
   internal_api: openmp
    num_threads: 12
         prefix: libomp
       filepath: /Users/y.korobko/mambaforge/envs/sklearn-env/lib/libomp.dylib
        version: None
```

COMMENT:
Before to make any enhancement, we can fix the documentation mentioning that a `ndarray` is returned.

Further enhancements related to this function are discussed in #27433

Labels: Documentation
DOC Improve description of the Nystroem method in the user guide

### Describe the issue linked to the documentation

We currently have a rather short and shallow description of the [Nystroem Method for Kernel Approximation](https://scikit-learn.org/stable/modules/kernel_approximation.html#nystroem-method-for-kernel-approximation).

### Suggest a potential alternative/fix

I think we can expand the user guide to first include an intuitive explanation of the method, and then a more mathematical explanation of the implementation, as done in [this stackexchange post](https://stats.stackexchange.com/questions/261149/nystroem-method-for-kernel-approximation) by @djsutherland.

COMMENT:
For further reference:
[wiki section](https://en.wikipedia.org/wiki/Low-rank_matrix_approximations#Nystr%C3%B6m_approximation) 
[this quite nice but heavy blogpost](https://maelfabien.github.io/machinelearning/largescale/#iv-nystr%C3%B6m-approximation)

Labels: Documentation, module:kernel_approximation
DOC Improve description of the Nystroem method in the user guide

### Describe the issue linked to the documentation

We currently have a rather short and shallow description of the [Nystroem Method for Kernel Approximation](https://scikit-learn.org/stable/modules/kernel_approximation.html#nystroem-method-for-kernel-approximation).

### Suggest a potential alternative/fix

I think we can expand the user guide to first include an intuitive explanation of the method, and then a more mathematical explanation of the implementation, as done in [this stackexchange post](https://stats.stackexchange.com/questions/261149/nystroem-method-for-kernel-approximation) by @djsutherland.

COMMENT:
I can take it if nobody else will

Labels: Documentation, module:kernel_approximation
DOC Improve description of the Nystroem method in the user guide

### Describe the issue linked to the documentation

We currently have a rather short and shallow description of the [Nystroem Method for Kernel Approximation](https://scikit-learn.org/stable/modules/kernel_approximation.html#nystroem-method-for-kernel-approximation).

### Suggest a potential alternative/fix

I think we can expand the user guide to first include an intuitive explanation of the method, and then a more mathematical explanation of the implementation, as done in [this stackexchange post](https://stats.stackexchange.com/questions/261149/nystroem-method-for-kernel-approximation) by @djsutherland.

COMMENT:
Feel free to do so @glevv :)

Labels: Documentation, module:kernel_approximation
DOC Improve description of the Nystroem method in the user guide

### Describe the issue linked to the documentation

We currently have a rather short and shallow description of the [Nystroem Method for Kernel Approximation](https://scikit-learn.org/stable/modules/kernel_approximation.html#nystroem-method-for-kernel-approximation).

### Suggest a potential alternative/fix

I think we can expand the user guide to first include an intuitive explanation of the method, and then a more mathematical explanation of the implementation, as done in [this stackexchange post](https://stats.stackexchange.com/questions/261149/nystroem-method-for-kernel-approximation) by @djsutherland.

COMMENT:
I think I need a reviewer

Labels: Documentation, module:kernel_approximation
Building from source fails using conda instructions

### Describe the bug

I am trying to build the development version of scikit-learn in Windows Subsystem for Linux running Ubuntu 20.04.5 and conda 23.7.2.

After creating the environment, building scikit-learn fails as cython is not found for some reason.

### Steps/Code to Reproduce

```shell
conda create -n sklearn-dev -c conda-forge python numpy scipy cython joblib threadpoolctl pytest compilers
conda activate sklearn-dev
pip install -v --no-use-pep517 --no-build-isolation -e .
``` 


### Expected Results

A succesful build.

### Actual Results

`ModuleNotFoundError: Please install Cython with a version >= 0.29.33 in order to build a scikit-learn from source.`

### Versions

```shell
Ubuntu 20.04.5 LTS
conda 23.7.2
```

COMMENT:
Could you close and reopen a terminal between the step 1 and step 2. Since `compilers` will modify some environment variables, it can happen that those don't take effect.

I tried on my linux machine and the above command are working as expected.

Labels: Documentation
Building from source fails using conda instructions

### Describe the bug

I am trying to build the development version of scikit-learn in Windows Subsystem for Linux running Ubuntu 20.04.5 and conda 23.7.2.

After creating the environment, building scikit-learn fails as cython is not found for some reason.

### Steps/Code to Reproduce

```shell
conda create -n sklearn-dev -c conda-forge python numpy scipy cython joblib threadpoolctl pytest compilers
conda activate sklearn-dev
pip install -v --no-use-pep517 --no-build-isolation -e .
``` 


### Expected Results

A succesful build.

### Actual Results

`ModuleNotFoundError: Please install Cython with a version >= 0.29.33 in order to build a scikit-learn from source.`

### Versions

```shell
Ubuntu 20.04.5 LTS
conda 23.7.2
```

COMMENT:
Thank you @glemaitre , that worked. Is this worth being noted in the docs? Another minor issue I faced before was that the dependencies could not be resolved using a 2 years old conda installation (conda 4.x).

Labels: Documentation
Building from source fails using conda instructions

### Describe the bug

I am trying to build the development version of scikit-learn in Windows Subsystem for Linux running Ubuntu 20.04.5 and conda 23.7.2.

After creating the environment, building scikit-learn fails as cython is not found for some reason.

### Steps/Code to Reproduce

```shell
conda create -n sklearn-dev -c conda-forge python numpy scipy cython joblib threadpoolctl pytest compilers
conda activate sklearn-dev
pip install -v --no-use-pep517 --no-build-isolation -e .
``` 


### Expected Results

A succesful build.

### Actual Results

`ModuleNotFoundError: Please install Cython with a version >= 0.29.33 in order to build a scikit-learn from source.`

### Versions

```shell
Ubuntu 20.04.5 LTS
conda 23.7.2
```

COMMENT:
Indeed, we could mention. I know it was the case in MacOS in some cases. Regarding the 2 years old conda, I would say that we cannot track so much those. We always put ourselves in a fresh install. But I am not surprise that the compilers package could not be found.

Labels: Documentation
Building from source fails using conda instructions

### Describe the bug

I am trying to build the development version of scikit-learn in Windows Subsystem for Linux running Ubuntu 20.04.5 and conda 23.7.2.

After creating the environment, building scikit-learn fails as cython is not found for some reason.

### Steps/Code to Reproduce

```shell
conda create -n sklearn-dev -c conda-forge python numpy scipy cython joblib threadpoolctl pytest compilers
conda activate sklearn-dev
pip install -v --no-use-pep517 --no-build-isolation -e .
``` 


### Expected Results

A succesful build.

### Actual Results

`ModuleNotFoundError: Please install Cython with a version >= 0.29.33 in order to build a scikit-learn from source.`

### Versions

```shell
Ubuntu 20.04.5 LTS
conda 23.7.2
```

COMMENT:
> Indeed, we could mention. I know it was the case in MacOS in some cases. Regarding the 2 years old conda, I would say that we cannot track so much those. We always put ourselves in a fresh install. But I am not surprise that the compilers package could not be found.

Allright, will try that out as my first contribution. :)

Labels: Documentation
RFC feature subsampling for tree based models

### Background
Currently, our (mostly tree-based) models supporting feature subsampling via `max_features` are:
- `DecisionTreeClassifier`, `DecisionTreeRegressor`
- `ExtraTreeClassifier`, `ExtraTreeRegressor`
- `BaggingClassifier`, `BaggingRegressor` 
- `ExtraTreesClassifier`, `ExtraTreesRegressor`
- `GradientBoostingClassifier`, `GradientBoostingRegressor`
- `IsolationForest`
- `RandomForestClassifier`, `RandomForestRegressor`

Soon, `HistGradientBoostingClassifier` and `HistGradientBoostingRegressor` will have feature subsampling, too, see #27139.

### Problem Statement
`max_features` can be a float between 0 and 1. In this case, it means that a **fraction** of features is randomly selected in each split. It can also be an integer, in which case it specifies the **number** of features selected in each split.
This means that `1` and `1.` have very different effects! See https://github.com/scikit-learn/scikit-learn/pull/27139#pullrequestreview-1619443083.

Also note that for most estimators the default is `max_features=None` or `1.0` meaning no subsampling. The notable exception is `RandomForestClassifier` and `ExtraTreesClassifier` with default `max_features="sqrt"` (which makes sense), but the corresponding regressors still use all features (so no rf but bagged trees). This was discussed in detail in #20111.

Additionally, for most estimator it means subsampling per tree split/node. For BaggingC/R models and IsolationForest it means per model/iteration.

### Proposal
Accompanied with a proper deprecation strategy, we could add different arguments, one for specifying fractions, one for integer numbers.

### Decision Options
#### A Do we want to change the current situation?
#### B Which names shall the new arguments have?
##### B.1
We could keep `max_features` for number of features and add only one new argument for fractions. Or we can actually add 2 new arguments and deprecate max_features completely.


##### B.2
Possible names for fractions, see also https://github.com/scikit-learn/scikit-learn/pull/27139#issuecomment-1715216957:
- `colsample_bynode`, same name as in XGBoost and LightGBM
- `feature_(sub)sample_per_split`
- `feature_fraction_per_split`

@scikit-learn/core-devs

COMMENT:
I personally don't see an issue with the current semantics of the parameter. So I wouldn't bother changing or adding a new parameter.

Labels: RFC
RFC feature subsampling for tree based models

### Background
Currently, our (mostly tree-based) models supporting feature subsampling via `max_features` are:
- `DecisionTreeClassifier`, `DecisionTreeRegressor`
- `ExtraTreeClassifier`, `ExtraTreeRegressor`
- `BaggingClassifier`, `BaggingRegressor` 
- `ExtraTreesClassifier`, `ExtraTreesRegressor`
- `GradientBoostingClassifier`, `GradientBoostingRegressor`
- `IsolationForest`
- `RandomForestClassifier`, `RandomForestRegressor`

Soon, `HistGradientBoostingClassifier` and `HistGradientBoostingRegressor` will have feature subsampling, too, see #27139.

### Problem Statement
`max_features` can be a float between 0 and 1. In this case, it means that a **fraction** of features is randomly selected in each split. It can also be an integer, in which case it specifies the **number** of features selected in each split.
This means that `1` and `1.` have very different effects! See https://github.com/scikit-learn/scikit-learn/pull/27139#pullrequestreview-1619443083.

Also note that for most estimators the default is `max_features=None` or `1.0` meaning no subsampling. The notable exception is `RandomForestClassifier` and `ExtraTreesClassifier` with default `max_features="sqrt"` (which makes sense), but the corresponding regressors still use all features (so no rf but bagged trees). This was discussed in detail in #20111.

Additionally, for most estimator it means subsampling per tree split/node. For BaggingC/R models and IsolationForest it means per model/iteration.

### Proposal
Accompanied with a proper deprecation strategy, we could add different arguments, one for specifying fractions, one for integer numbers.

### Decision Options
#### A Do we want to change the current situation?
#### B Which names shall the new arguments have?
##### B.1
We could keep `max_features` for number of features and add only one new argument for fractions. Or we can actually add 2 new arguments and deprecate max_features completely.


##### B.2
Possible names for fractions, see also https://github.com/scikit-learn/scikit-learn/pull/27139#issuecomment-1715216957:
- `colsample_bynode`, same name as in XGBoost and LightGBM
- `feature_(sub)sample_per_split`
- `feature_fraction_per_split`

@scikit-learn/core-devs

COMMENT:
An interesting thing to know, that I don't know how to find out, is how often someone uses `max_features=1.` when they meant to use `max_features=1`. This seems like the only(?) case where a user would not get what they wanted and it might be a bit tricky to track down why.

If this mistake doesn't happen a lot, then maybe nothing is broken/needs fixing/changing?

The code in scikit-learn itself would be simpler if we changed this, but I'm not sure if making our lives easier is a big enough reason to change?

Labels: RFC