Talip7/scikit-learn-issues-multilabel huggingface dataset:
KernelDensity incorrect handling of bandwidth

### Describe the bug

I was using kernel density estimator
https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KernelDensity.html
using 'silverman' or 'scott' as the bandwidth argument. Then I found that the bandwidth automatically adjusted by the algorithm is independent of the actual scale of the dataset. In fact, I was shocked to find that the calculation of a bandwidth in https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/neighbors/_kde.py for 'silverman' and 'scott' does not check the scales of data at all. 

Suppose I fit the model `kde` to some 2D data `X` and get the bandwidth as `kde.bandwidth_`.
Next, I fit the model `kde` to the same 2D data `X` but with all elements multiplied by, say, 20 and get the bandwidth as `kde.bandwidth_`. 
I found that these two values of `kde.bandwidth_` are equal (it is calculated from the shape of `X`, see the source code). But obviously they should differ by a factor of 20 if the bandwidth is really computed in a truly adaptive manner.

For your reference, I want to mention that scipy's KDE https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gaussian_kde.html calculates the covariance of data to extract the scale of data. I think this is the right thing to do.

Note that if the bandwidth is incorrect, everything else is incorrect too, including probablities of samples, etc.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.neighbors import KernelDensity
X = np.random.randn(1000, 2)
kde = KernelDensity(bandwidth='scott')
kde.fit(X)
print(kde.bandwidth_)

kde.fit(X * 20)
print(kde.bandwidth_)
```

### Expected Results

Different bandwidths for data sets with different scales.

### Actual Results

0.31622776601683794
0.31622776601683794

### Versions

```shell
1.2.1
```

COMMENT:
Thank you for opening this! I agree this is a bug. Here is the same plot using SciPy's kernel function that estimates the correct bandwidth:

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm
from scipy import stats

N = 200
rng = np.random.default_rng(42)

X = np.concatenate(
    (rng.normal(0, 5, int(0.3 * N)), rng.normal(15, 5, int(0.7 * N)))
)

X_plot = np.linspace(-15, 30, 1000)
true_dens = 0.3 * norm(loc=0, scale=5).pdf(X_plot) + 0.7 * norm(loc=15, scale=5).pdf(X_plot)

fig, ax = plt.subplots()
ax.fill(X_plot, true_dens, fc="black", alpha=0.2, label="input distribution")

colors = ["navy", "red"]
methods = ["silverman", "scott"]
lw = 2

for color, method in zip(colors, methods):
    kde = stats.gaussian_kde(X, bw_method=method)
    density = kde(X_plot)
    ax.plot(
        X_plot,
        density,
        color=color,
        lw=lw,
        linestyle="-",
        label="method = '{0}'".format(method),
    )

ax.legend(loc="upper left")
```

![Screenshot 2023-02-24 at 12 19 11 PM](https://user-images.githubusercontent.com/5402633/221245383-cb21ad47-712d-4d38-8246-bf12d8596df5.jpg)

Labels: Bug, module:neighbors
KernelDensity incorrect handling of bandwidth

### Describe the bug

I was using kernel density estimator
https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KernelDensity.html
using 'silverman' or 'scott' as the bandwidth argument. Then I found that the bandwidth automatically adjusted by the algorithm is independent of the actual scale of the dataset. In fact, I was shocked to find that the calculation of a bandwidth in https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/neighbors/_kde.py for 'silverman' and 'scott' does not check the scales of data at all. 

Suppose I fit the model `kde` to some 2D data `X` and get the bandwidth as `kde.bandwidth_`.
Next, I fit the model `kde` to the same 2D data `X` but with all elements multiplied by, say, 20 and get the bandwidth as `kde.bandwidth_`. 
I found that these two values of `kde.bandwidth_` are equal (it is calculated from the shape of `X`, see the source code). But obviously they should differ by a factor of 20 if the bandwidth is really computed in a truly adaptive manner.

For your reference, I want to mention that scipy's KDE https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gaussian_kde.html calculates the covariance of data to extract the scale of data. I think this is the right thing to do.

Note that if the bandwidth is incorrect, everything else is incorrect too, including probablities of samples, etc.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.neighbors import KernelDensity
X = np.random.randn(1000, 2)
kde = KernelDensity(bandwidth='scott')
kde.fit(X)
print(kde.bandwidth_)

kde.fit(X * 20)
print(kde.bandwidth_)
```

### Expected Results

Different bandwidths for data sets with different scales.

### Actual Results

0.31622776601683794
0.31622776601683794

### Versions

```shell
1.2.1
```

COMMENT:
@thomasjpfan could you report the `factor` given by SciPy with our `kernel.bandwidth_`.
I think they will be the same. I am not sure that we should alter the bandwidth directly but rather scale it when passing it to the parameter `h` to the `tree_`?

Labels: Bug, module:neighbors
KernelDensity incorrect handling of bandwidth

### Describe the bug

I was using kernel density estimator
https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KernelDensity.html
using 'silverman' or 'scott' as the bandwidth argument. Then I found that the bandwidth automatically adjusted by the algorithm is independent of the actual scale of the dataset. In fact, I was shocked to find that the calculation of a bandwidth in https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/neighbors/_kde.py for 'silverman' and 'scott' does not check the scales of data at all. 

Suppose I fit the model `kde` to some 2D data `X` and get the bandwidth as `kde.bandwidth_`.
Next, I fit the model `kde` to the same 2D data `X` but with all elements multiplied by, say, 20 and get the bandwidth as `kde.bandwidth_`. 
I found that these two values of `kde.bandwidth_` are equal (it is calculated from the shape of `X`, see the source code). But obviously they should differ by a factor of 20 if the bandwidth is really computed in a truly adaptive manner.

For your reference, I want to mention that scipy's KDE https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gaussian_kde.html calculates the covariance of data to extract the scale of data. I think this is the right thing to do.

Note that if the bandwidth is incorrect, everything else is incorrect too, including probablities of samples, etc.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.neighbors import KernelDensity
X = np.random.randn(1000, 2)
kde = KernelDensity(bandwidth='scott')
kde.fit(X)
print(kde.bandwidth_)

kde.fit(X * 20)
print(kde.bandwidth_)
```

### Expected Results

Different bandwidths for data sets with different scales.

### Actual Results

0.31622776601683794
0.31622776601683794

### Versions

```shell
1.2.1
```

COMMENT:
Thank you so much for nice figures :-) @thomasjpfan

In addition to the data density issue, I'm also concerned with possible anisotropy of data. 
Currently sklearn's KernelDensity implementation seems to asssume an isotropic kernel, while scipy's gaussian kernel seems to allow for an anisotropic covariance matrix (please correct me if I'm wrong). 
In fact, data points (in 2D, for example) can have wildly different scales of nearest neighbor distance along X axis and Y axis, as shown in the sample data below. 
So I suppose the desiderata are to reflect the actual density of data points in the bandwidth, and to allow for anisotropy of kernels (at least for gaussian kernel). 
```python
mean = [0, 0]
cov = [[4000, 0], [0, 1]]
np.random.seed(12)
data = np.random.multivariate_normal(mean, cov, 220) @ np.array([[1., 0.04], [-0.04, 1.]])
plt.tick_params(labelsize=17)
plt.scatter(data[:, 0], data[:, 1], marker='^', s=33, c='cyan', edgecolors='grey')
plt.show()
```
<img src="https://user-images.githubusercontent.com/52037101/221293380-c28450c7-53b5-4464-a4d9-c8a5ef66cbcb.png" width="512"/>

Labels: Bug, module:neighbors
KernelDensity incorrect handling of bandwidth

### Describe the bug

I was using kernel density estimator
https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KernelDensity.html
using 'silverman' or 'scott' as the bandwidth argument. Then I found that the bandwidth automatically adjusted by the algorithm is independent of the actual scale of the dataset. In fact, I was shocked to find that the calculation of a bandwidth in https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/neighbors/_kde.py for 'silverman' and 'scott' does not check the scales of data at all. 

Suppose I fit the model `kde` to some 2D data `X` and get the bandwidth as `kde.bandwidth_`.
Next, I fit the model `kde` to the same 2D data `X` but with all elements multiplied by, say, 20 and get the bandwidth as `kde.bandwidth_`. 
I found that these two values of `kde.bandwidth_` are equal (it is calculated from the shape of `X`, see the source code). But obviously they should differ by a factor of 20 if the bandwidth is really computed in a truly adaptive manner.

For your reference, I want to mention that scipy's KDE https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gaussian_kde.html calculates the covariance of data to extract the scale of data. I think this is the right thing to do.

Note that if the bandwidth is incorrect, everything else is incorrect too, including probablities of samples, etc.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.neighbors import KernelDensity
X = np.random.randn(1000, 2)
kde = KernelDensity(bandwidth='scott')
kde.fit(X)
print(kde.bandwidth_)

kde.fit(X * 20)
print(kde.bandwidth_)
```

### Expected Results

Different bandwidths for data sets with different scales.

### Actual Results

0.31622776601683794
0.31622776601683794

### Versions

```shell
1.2.1
```

COMMENT:
ref https://github.com/scikit-learn/scikit-learn/issues/26658#issuecomment-1617403380 may maintainers take a look? @thomasjpfan Transforming data seems to be reasonable, though maybe it should be done in the binary tree code when calling `kernel_density` because this is a public method.

@TaTKSM at the bottom of the linked comment above there is a 2D example. Though different from yours I think it is a similar case as https://github.com/scikit-learn/scikit-learn/issues/25623#issuecomment-1444505841. Also I think the covariance matrix approach can be used not only for Gaussian kernel. In general

$$
\frac{1}{N|H|}\sum_{i=1}^N\mathcal{K}\left(H^{-1}(x-x^i)\right),
$$

where $H$ is the bandwidth matrix chosen proportional to $\Sigma^{1/2}$, square root of the covariance matrix. $\mathcal{K}$ can be any kernel here I believe.

Labels: Bug, module:neighbors
Simplify decision tree removing redundant decisions

### Describe the workflow you want to enable

Description: Add a new method simplify() to the decision tree Class that returns a simplified version of the decision tree by pruning redundant leaves that do not add new decision paths. This simplification method will create a new instance of the tree with fewer nodes, allowing users to obtain a more concise and interpretable tree.

Motivation: Decision trees are known for their ability to provide easily interpretable models, but sometimes they can grow to a size that makes interpretation challenging. A simplified version of the decision tree will make it easier for users to understand the model, especially in situations where the tree has a large number of redundant leaves.

Example:
![image](https://user-images.githubusercontent.com/11466701/218794306-eac085c6-2426-4e1f-aaf9-d563b2561922.png)
Can be simplified to
![image](https://user-images.githubusercontent.com/11466701/218795262-0a625e63-633d-44e0-97b2-653b0b8fc27a.png)

Related Work: There are related issues in [Scikit-learn #10810](https://github.com/scikit-learn/scikit-learn/issues/10810), [IBM Taxinomitis #226](https://github.com/IBM/taxinomitis/issues/226) and [Stack Overfflow](https://stackoverflow.com/questions/51397109/prune-unnecessary-leaves-in-sklearn-decisiontreeclassifier) that point to the need for decision tree simplification. Scikit-learn has added post-pruning methods based on cost, as far as I know, they do not achieve the same level of simplification proposed in this feature.



### Describe your proposed solution

Proposed Solution: Add a simplify() method to the decision tree module that returns a simplified version of the tree. The method will prune redundant leaves that do not add new decision paths. The new tree will have fewer nodes and be easier to interpret.

Example Usage:
```Python
from sklearn.tree import DecisionTreeClassifier

clf = DecisionTreeClassifier()

...

clf.fit(X, y)

tree = clf.tree_
simplified_tree = tree.simplify()
```


### Describe alternatives you've considered, if relevant

An alternative way is to create ad hoc functions to achieve this simplification. But I believe this feature is important enough to be in the main repository

### Additional context

I can work on this feature request and send a PR, if approved by members.

COMMENT:
What is the reference for this pruning technique?

Labels: New Feature, module:tree, Needs Investigation
Improve the visibility of the projects governance

### Describe the issue linked to the documentation

When I navigate to https://scikit-learn.org/stable/governance.html#governance, I first got to scikit-learn.org -> More -> About Us, and then there is a link to the governance. This should be improved!

### Suggest a potential alternative/fix

_No response_

COMMENT:
I think this makes sense. If we want to reduce the number of clicks, then the navbar would be the more visible. I opened https://github.com/scikit-learn/scikit-learn/pull/25618 to place it there.

Labels: Documentation
Building from source fails on Linux systems with pre-installed Intel OpenMP

### Describe the issue linked to the documentation

On systems that have Intel compilers with OpenMP support (like `icx` and `icpc`), building scikit-learn from source fails with the following error.

`ImportError: libomp.so: cannot open shared object file: No such file or directory`

### Suggest a potential alternative/fix

Building scikit-learn from source uses `-fopenmp` to compile with OpenMP. However, Intel compilers require `-qopenmp` option to compile OpenMP applications. See Intel OpenMP documentation [here](https://www.intel.com/content/www/us/en/develop/documentation/cpp-compiler-developer-guide-and-reference/top/optimization-and-programming/openmp-support/openmp-library-support/use-the-openmp-libraries.html).

I was able to overcome the build failure by explicitly setting the `CPPFLAGS` environment variable as below.

`export CPPFLAGS="$CPPFLAGS -qopenmp"`

The documentation needs to direct users compiling with Intel compilers to set the environment variable before building from source.

COMMENT:
@thomasjpfan @jeremiedbb I don't recall the issue that we had such that we wanted to drop the ICC buid in the CI? Could you refresh my memory?

We could probably a section on the intel compiler in the following documentation: https://scikit-learn.org/stable/developers/advanced_installation.html#linux-compilers-from-the-system

Labels: Documentation
Building from source fails on Linux systems with pre-installed Intel OpenMP

### Describe the issue linked to the documentation

On systems that have Intel compilers with OpenMP support (like `icx` and `icpc`), building scikit-learn from source fails with the following error.

`ImportError: libomp.so: cannot open shared object file: No such file or directory`

### Suggest a potential alternative/fix

Building scikit-learn from source uses `-fopenmp` to compile with OpenMP. However, Intel compilers require `-qopenmp` option to compile OpenMP applications. See Intel OpenMP documentation [here](https://www.intel.com/content/www/us/en/develop/documentation/cpp-compiler-developer-guide-and-reference/top/optimization-and-programming/openmp-support/openmp-library-support/use-the-openmp-libraries.html).

I was able to overcome the build failure by explicitly setting the `CPPFLAGS` environment variable as below.

`export CPPFLAGS="$CPPFLAGS -qopenmp"`

The documentation needs to direct users compiling with Intel compilers to set the environment variable before building from source.

COMMENT:
We dropped the support for custom compilers (using ``--compiler=``) with the drop of distutils. It is still possible to pass a custom compiler through env vars like ``CC`` but in that case it also requires setting the appropriate flags for openmp through env vars as well. 

The documentation for custom compilers was completely removed when dropping distutils but I think that we can add back a small section about how to use custom compilers.

Labels: Documentation
Building from source fails on Linux systems with pre-installed Intel OpenMP

### Describe the issue linked to the documentation

On systems that have Intel compilers with OpenMP support (like `icx` and `icpc`), building scikit-learn from source fails with the following error.

`ImportError: libomp.so: cannot open shared object file: No such file or directory`

### Suggest a potential alternative/fix

Building scikit-learn from source uses `-fopenmp` to compile with OpenMP. However, Intel compilers require `-qopenmp` option to compile OpenMP applications. See Intel OpenMP documentation [here](https://www.intel.com/content/www/us/en/develop/documentation/cpp-compiler-developer-guide-and-reference/top/optimization-and-programming/openmp-support/openmp-library-support/use-the-openmp-libraries.html).

I was able to overcome the build failure by explicitly setting the `CPPFLAGS` environment variable as below.

`export CPPFLAGS="$CPPFLAGS -qopenmp"`

The documentation needs to direct users compiling with Intel compilers to set the environment variable before building from source.

COMMENT:
I agree with adding a generic section about adjusting `CPPFLAGS` and friends to enable custom compilers. I'll be careful about giving Intel specific instructions until we can verify on CI that those instructions work and tests pass.

Labels: Documentation
Building from source fails on Linux systems with pre-installed Intel OpenMP

### Describe the issue linked to the documentation

On systems that have Intel compilers with OpenMP support (like `icx` and `icpc`), building scikit-learn from source fails with the following error.

`ImportError: libomp.so: cannot open shared object file: No such file or directory`

### Suggest a potential alternative/fix

Building scikit-learn from source uses `-fopenmp` to compile with OpenMP. However, Intel compilers require `-qopenmp` option to compile OpenMP applications. See Intel OpenMP documentation [here](https://www.intel.com/content/www/us/en/develop/documentation/cpp-compiler-developer-guide-and-reference/top/optimization-and-programming/openmp-support/openmp-library-support/use-the-openmp-libraries.html).

I was able to overcome the build failure by explicitly setting the `CPPFLAGS` environment variable as below.

`export CPPFLAGS="$CPPFLAGS -qopenmp"`

The documentation needs to direct users compiling with Intel compilers to set the environment variable before building from source.

COMMENT:
I opened https://github.com/scikit-learn/scikit-learn/pull/25667

Labels: Documentation
Train-test-split for multilabel datasets

### Describe the workflow you want to enable

Train-test splits for 2-dimensional targets (i.e. multi-label datasets).
I saw couple of issues related to multi-label classification, but as far as I can tell, train-test-split has not been addressed there.

Related forum questions:
- https://datascience.stackexchange.com/questions/45174/how-to-use-sklearn-train-test-split-to-stratify-data-for-multi-label-classificat 
- https://stackoverflow.com/questions/65030825/multi-label-imbalanced-train-test-split

### Describe your proposed solution

See description. 

### Describe alternatives you've considered, if relevant

Use of one of the following packages:
- https://github.com/trent-b/iterative-stratification 
- https://github.com/scikit-multilearn/scikit-multilearn with iterative_stratification method: http://scikit.ml/api/skmultilearn.model_selection.iterative_stratification.html#module-skmultilearn.model_selection.iterative_stratification

### Additional context

_No response_

COMMENT:
Hi 👋! Thanks for opening this issue. Could you provide a code snippet with a toy dataset that illustrates what you'd like to do (and that doesn't work right now)?

Labels: New Feature, Needs Triage
Train-test-split for multilabel datasets

### Describe the workflow you want to enable

Train-test splits for 2-dimensional targets (i.e. multi-label datasets).
I saw couple of issues related to multi-label classification, but as far as I can tell, train-test-split has not been addressed there.

Related forum questions:
- https://datascience.stackexchange.com/questions/45174/how-to-use-sklearn-train-test-split-to-stratify-data-for-multi-label-classificat 
- https://stackoverflow.com/questions/65030825/multi-label-imbalanced-train-test-split

### Describe your proposed solution

See description. 

### Describe alternatives you've considered, if relevant

Use of one of the following packages:
- https://github.com/trent-b/iterative-stratification 
- https://github.com/scikit-multilearn/scikit-multilearn with iterative_stratification method: http://scikit.ml/api/skmultilearn.model_selection.iterative_stratification.html#module-skmultilearn.model_selection.iterative_stratification

### Additional context

_No response_

COMMENT:
Hi @betatim 
Apologies for not providing a minimal example, which furthermore proves me wrong, but I still deliver it here. It shows though that this actually works :) I think there must have been an issue with that there were a few combinations that only occured once (which does not occur when framing it as a multi-class problem).

```python
from sklearn.datasets import make_multilabel_classification
from sklearn.model_selection import train_test_split

X,y = make_multilabel_classification(n_samples=1000, n_features=10, n_classes=3, n_labels=2, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X,y,train_size=0.3, random_state=42, shuffle=True, stratify=y)
```

Labels: New Feature, Needs Triage