Talip7/scikit-learn-issues-multilabel huggingface dataset:
Non-metric MDS gives nonsense results on the Digits dataset

### Describe the bug

I am running metric and nonmetric MDS on the Digits dataset, and obtain nonsense results with `metric=False`. At the same time, my understanding is that non-metric MDS is more flexible and should not yield worse results compared to metric MDS. Is my understanding wrong, or does it suggest some problems with the non-metric MDS implementation?

### Steps/Code to Reproduce

```Python
%matplotlib notebook

import pylab as plt
import numpy as np
from sklearn.datasets import load_digits
from sklearn.manifold import MDS

digits = load_digits()
X, y = digits.data, digits.target

Z1 = MDS(n_components=2, n_init=1, random_state=42).fit_transform(X)
Z2 = MDS(n_components=2, n_init=1, random_state=42, metric=False).fit_transform(X)

titles = ['Metric MDS', 'Non-metric MDS']

fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(6, 3), layout='constrained')

for i,Z in enumerate([Z1, Z2]):
    ax = axs.ravel()[i]
    
    ax.set_aspect('equal', 'datalim')
    
    ind = np.random.permutation(X.shape[0])
    ax.scatter(*Z[ind].T, s=3, color=plt.cm.Dark2(y)[ind])

    ax.set_xticks([])
    ax.set_yticks([])
    for sp in ax.spines:
        ax.spines[sp].set_visible(False)
        
    ax.set_title(titles[i])
    
fig.savefig('digits-embed-mds-nonmetric.png', dpi=300)
```

### Expected Results

Non-metric MDS giving something sensible.

### Actual Results

![digits-embed-mds-nonmetric](https://github.com/scikit-learn/scikit-learn/assets/8970231/aa6af88a-75d0-4ef2-ab6e-6482c4df5ae8)


### Versions

```shell
System:
    python: 3.7.9 (default, Aug 31 2020, 12:42:55)  [GCC 7.3.0]
executable: /home/dmitry/anaconda3/bin/python
   machine: Linux-5.15.0-76-generic-x86_64-with-debian-bullseye-sid

Python dependencies:
          pip: 22.3.1
   setuptools: 65.6.3
      sklearn: 0.24.1
        numpy: 1.21.5
        scipy: 1.7.3
       Cython: 0.29.32
       pandas: 1.3.5
   matplotlib: 3.5.2
       joblib: 1.1.1
threadpoolctl: 2.2.0

Built with OpenMP: True
```

COMMENT:
May be related to https://github.com/scikit-learn/scikit-learn/issues/18933 but I'm not sure. CC @henrymartin1.

Labels: Bug, Needs Investigation
Non-metric MDS gives nonsense results on the Digits dataset

### Describe the bug

I am running metric and nonmetric MDS on the Digits dataset, and obtain nonsense results with `metric=False`. At the same time, my understanding is that non-metric MDS is more flexible and should not yield worse results compared to metric MDS. Is my understanding wrong, or does it suggest some problems with the non-metric MDS implementation?

### Steps/Code to Reproduce

```Python
%matplotlib notebook

import pylab as plt
import numpy as np
from sklearn.datasets import load_digits
from sklearn.manifold import MDS

digits = load_digits()
X, y = digits.data, digits.target

Z1 = MDS(n_components=2, n_init=1, random_state=42).fit_transform(X)
Z2 = MDS(n_components=2, n_init=1, random_state=42, metric=False).fit_transform(X)

titles = ['Metric MDS', 'Non-metric MDS']

fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(6, 3), layout='constrained')

for i,Z in enumerate([Z1, Z2]):
    ax = axs.ravel()[i]
    
    ax.set_aspect('equal', 'datalim')
    
    ind = np.random.permutation(X.shape[0])
    ax.scatter(*Z[ind].T, s=3, color=plt.cm.Dark2(y)[ind])

    ax.set_xticks([])
    ax.set_yticks([])
    for sp in ax.spines:
        ax.spines[sp].set_visible(False)
        
    ax.set_title(titles[i])
    
fig.savefig('digits-embed-mds-nonmetric.png', dpi=300)
```

### Expected Results

Non-metric MDS giving something sensible.

### Actual Results

![digits-embed-mds-nonmetric](https://github.com/scikit-learn/scikit-learn/assets/8970231/aa6af88a-75d0-4ef2-ab6e-6482c4df5ae8)


### Versions

```shell
System:
    python: 3.7.9 (default, Aug 31 2020, 12:42:55)  [GCC 7.3.0]
executable: /home/dmitry/anaconda3/bin/python
   machine: Linux-5.15.0-76-generic-x86_64-with-debian-bullseye-sid

Python dependencies:
          pip: 22.3.1
   setuptools: 65.6.3
      sklearn: 0.24.1
        numpy: 1.21.5
        scipy: 1.7.3
       Cython: 0.29.32
       pandas: 1.3.5
   matplotlib: 3.5.2
       joblib: 1.1.1
threadpoolctl: 2.2.0

Built with OpenMP: True
```

COMMENT:
> What do you think @Micky774? To me the above is evidence that the optimisation is not working correctly.

Yeah at this point it definitely seems like something is wrong. I unfortunately don't have much extra bandwidth for this at the moment, but I will offer a review if anyone else would like to take a stab at this.

> May be related to https://github.com/scikit-learn/scikit-learn/issues/18933 but I'm not sure

I would like to see this issue worked on first, then revisit this instance to determine if the bug persists.

Thank you for taking the time to advocate for this issue :)

Labels: Bug, Needs Investigation
Non-metric MDS gives nonsense results on the Digits dataset

### Describe the bug

I am running metric and nonmetric MDS on the Digits dataset, and obtain nonsense results with `metric=False`. At the same time, my understanding is that non-metric MDS is more flexible and should not yield worse results compared to metric MDS. Is my understanding wrong, or does it suggest some problems with the non-metric MDS implementation?

### Steps/Code to Reproduce

```Python
%matplotlib notebook

import pylab as plt
import numpy as np
from sklearn.datasets import load_digits
from sklearn.manifold import MDS

digits = load_digits()
X, y = digits.data, digits.target

Z1 = MDS(n_components=2, n_init=1, random_state=42).fit_transform(X)
Z2 = MDS(n_components=2, n_init=1, random_state=42, metric=False).fit_transform(X)

titles = ['Metric MDS', 'Non-metric MDS']

fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(6, 3), layout='constrained')

for i,Z in enumerate([Z1, Z2]):
    ax = axs.ravel()[i]
    
    ax.set_aspect('equal', 'datalim')
    
    ind = np.random.permutation(X.shape[0])
    ax.scatter(*Z[ind].T, s=3, color=plt.cm.Dark2(y)[ind])

    ax.set_xticks([])
    ax.set_yticks([])
    for sp in ax.spines:
        ax.spines[sp].set_visible(False)
        
    ax.set_title(titles[i])
    
fig.savefig('digits-embed-mds-nonmetric.png', dpi=300)
```

### Expected Results

Non-metric MDS giving something sensible.

### Actual Results

![digits-embed-mds-nonmetric](https://github.com/scikit-learn/scikit-learn/assets/8970231/aa6af88a-75d0-4ef2-ab6e-6482c4df5ae8)


### Versions

```shell
System:
    python: 3.7.9 (default, Aug 31 2020, 12:42:55)  [GCC 7.3.0]
executable: /home/dmitry/anaconda3/bin/python
   machine: Linux-5.15.0-76-generic-x86_64-with-debian-bullseye-sid

Python dependencies:
          pip: 22.3.1
   setuptools: 65.6.3
      sklearn: 0.24.1
        numpy: 1.21.5
        scipy: 1.7.3
       Cython: 0.29.32
       pandas: 1.3.5
   matplotlib: 3.5.2
       joblib: 1.1.1
threadpoolctl: 2.2.0

Built with OpenMP: True
```

COMMENT:
Should be fixed in my https://github.com/scikit-learn/scikit-learn/pull/30514.

Labels: Bug, Needs Investigation
Kernel density estimation on (semi-)bounded domains

### Describe the workflow you want to enable

Kernel density estimates for bounded data are biased near the boundary because probability mass "spills out of the domain". It would be great to add a boundary correction to `KernelDensity`. For example, estimating the uniform density on the unit interval is severely biased near the boundaries.

```python
from matplotlib import pyplot as plt
import numpy as np
from sklearn.neighbors import KernelDensity


X = np.random.uniform(0, 1, (10_000, 1))
kde = KernelDensity(bandwidth="silverman").fit(X)
x = np.linspace(0, 1)
plt.plot(x, np.exp(kde.score_samples(x[:, None])))
```

![image](https://github.com/scikit-learn/scikit-learn/assets/966348/dc213fc8-7475-4f82-82e3-d3b883ee44e7)


### Describe your proposed solution

Reflection methods reflect samples to be scored at the boundary such that the probability mass outside the support is "moved back" into the domain. Such a reflection could be implemented as follows.

1. Add a `bounds` parameter to the `KernelDensity` constructor with shape `(n_features, 2)` comprising the lower and upper bound in each dimension of the feature space.
2. Raise a `ValueError` in `fit` if one or more samples are outside the specified bounds.
3. Reflect samples to be scored at the boundaries in `score_samples` and combine contributions from all reflections using `logsumexp`.

Alternatively, a new estimator `BoundedKernelDensity` could inherit from `KernelDensity` to keep the `KernelDensity` API from ballooning. Would either be of interest for inclusion in scikit-learn?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Can I work on this Issue?

Labels: New Feature
Kernel density estimation on (semi-)bounded domains

### Describe the workflow you want to enable

Kernel density estimates for bounded data are biased near the boundary because probability mass "spills out of the domain". It would be great to add a boundary correction to `KernelDensity`. For example, estimating the uniform density on the unit interval is severely biased near the boundaries.

```python
from matplotlib import pyplot as plt
import numpy as np
from sklearn.neighbors import KernelDensity


X = np.random.uniform(0, 1, (10_000, 1))
kde = KernelDensity(bandwidth="silverman").fit(X)
x = np.linspace(0, 1)
plt.plot(x, np.exp(kde.score_samples(x[:, None])))
```

![image](https://github.com/scikit-learn/scikit-learn/assets/966348/dc213fc8-7475-4f82-82e3-d3b883ee44e7)


### Describe your proposed solution

Reflection methods reflect samples to be scored at the boundary such that the probability mass outside the support is "moved back" into the domain. Such a reflection could be implemented as follows.

1. Add a `bounds` parameter to the `KernelDensity` constructor with shape `(n_features, 2)` comprising the lower and upper bound in each dimension of the feature space.
2. Raise a `ValueError` in `fit` if one or more samples are outside the specified bounds.
3. Reflect samples to be scored at the boundaries in `score_samples` and combine contributions from all reflections using `logsumexp`.

Alternatively, a new estimator `BoundedKernelDensity` could inherit from `KernelDensity` to keep the `KernelDensity` API from ballooning. Would either be of interest for inclusion in scikit-learn?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@Kaustbh the first step for this issue is to come to a decision about including it or not (this is what the "needs triage" label means).

Labels: New Feature
Kernel density estimation on (semi-)bounded domains

### Describe the workflow you want to enable

Kernel density estimates for bounded data are biased near the boundary because probability mass "spills out of the domain". It would be great to add a boundary correction to `KernelDensity`. For example, estimating the uniform density on the unit interval is severely biased near the boundaries.

```python
from matplotlib import pyplot as plt
import numpy as np
from sklearn.neighbors import KernelDensity


X = np.random.uniform(0, 1, (10_000, 1))
kde = KernelDensity(bandwidth="silverman").fit(X)
x = np.linspace(0, 1)
plt.plot(x, np.exp(kde.score_samples(x[:, None])))
```

![image](https://github.com/scikit-learn/scikit-learn/assets/966348/dc213fc8-7475-4f82-82e3-d3b883ee44e7)


### Describe your proposed solution

Reflection methods reflect samples to be scored at the boundary such that the probability mass outside the support is "moved back" into the domain. Such a reflection could be implemented as follows.

1. Add a `bounds` parameter to the `KernelDensity` constructor with shape `(n_features, 2)` comprising the lower and upper bound in each dimension of the feature space.
2. Raise a `ValueError` in `fit` if one or more samples are outside the specified bounds.
3. Reflect samples to be scored at the boundaries in `score_samples` and combine contributions from all reflections using `logsumexp`.

Alternatively, a new estimator `BoundedKernelDensity` could inherit from `KernelDensity` to keep the `KernelDensity` API from ballooning. Would either be of interest for inclusion in scikit-learn?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Okay.

Labels: New Feature
Kernel density estimation on (semi-)bounded domains

### Describe the workflow you want to enable

Kernel density estimates for bounded data are biased near the boundary because probability mass "spills out of the domain". It would be great to add a boundary correction to `KernelDensity`. For example, estimating the uniform density on the unit interval is severely biased near the boundaries.

```python
from matplotlib import pyplot as plt
import numpy as np
from sklearn.neighbors import KernelDensity


X = np.random.uniform(0, 1, (10_000, 1))
kde = KernelDensity(bandwidth="silverman").fit(X)
x = np.linspace(0, 1)
plt.plot(x, np.exp(kde.score_samples(x[:, None])))
```

![image](https://github.com/scikit-learn/scikit-learn/assets/966348/dc213fc8-7475-4f82-82e3-d3b883ee44e7)


### Describe your proposed solution

Reflection methods reflect samples to be scored at the boundary such that the probability mass outside the support is "moved back" into the domain. Such a reflection could be implemented as follows.

1. Add a `bounds` parameter to the `KernelDensity` constructor with shape `(n_features, 2)` comprising the lower and upper bound in each dimension of the feature space.
2. Raise a `ValueError` in `fit` if one or more samples are outside the specified bounds.
3. Reflect samples to be scored at the boundaries in `score_samples` and combine contributions from all reflections using `logsumexp`.

Alternatively, a new estimator `BoundedKernelDensity` could inherit from `KernelDensity` to keep the `KernelDensity` API from ballooning. Would either be of interest for inclusion in scikit-learn?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
It seems to be worth the effort. By default, we can keep the current behaviour.

Labels: New Feature
Kernel density estimation on (semi-)bounded domains

### Describe the workflow you want to enable

Kernel density estimates for bounded data are biased near the boundary because probability mass "spills out of the domain". It would be great to add a boundary correction to `KernelDensity`. For example, estimating the uniform density on the unit interval is severely biased near the boundaries.

```python
from matplotlib import pyplot as plt
import numpy as np
from sklearn.neighbors import KernelDensity


X = np.random.uniform(0, 1, (10_000, 1))
kde = KernelDensity(bandwidth="silverman").fit(X)
x = np.linspace(0, 1)
plt.plot(x, np.exp(kde.score_samples(x[:, None])))
```

![image](https://github.com/scikit-learn/scikit-learn/assets/966348/dc213fc8-7475-4f82-82e3-d3b883ee44e7)


### Describe your proposed solution

Reflection methods reflect samples to be scored at the boundary such that the probability mass outside the support is "moved back" into the domain. Such a reflection could be implemented as follows.

1. Add a `bounds` parameter to the `KernelDensity` constructor with shape `(n_features, 2)` comprising the lower and upper bound in each dimension of the feature space.
2. Raise a `ValueError` in `fit` if one or more samples are outside the specified bounds.
3. Reflect samples to be scored at the boundaries in `score_samples` and combine contributions from all reflections using `logsumexp`.

Alternatively, a new estimator `BoundedKernelDensity` could inherit from `KernelDensity` to keep the `KernelDensity` API from ballooning. Would either be of interest for inclusion in scikit-learn?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@tillahoffmann Would you still be interested at implementing the feature.

Labels: New Feature
Kernel density estimation on (semi-)bounded domains

### Describe the workflow you want to enable

Kernel density estimates for bounded data are biased near the boundary because probability mass "spills out of the domain". It would be great to add a boundary correction to `KernelDensity`. For example, estimating the uniform density on the unit interval is severely biased near the boundaries.

```python
from matplotlib import pyplot as plt
import numpy as np
from sklearn.neighbors import KernelDensity


X = np.random.uniform(0, 1, (10_000, 1))
kde = KernelDensity(bandwidth="silverman").fit(X)
x = np.linspace(0, 1)
plt.plot(x, np.exp(kde.score_samples(x[:, None])))
```

![image](https://github.com/scikit-learn/scikit-learn/assets/966348/dc213fc8-7475-4f82-82e3-d3b883ee44e7)


### Describe your proposed solution

Reflection methods reflect samples to be scored at the boundary such that the probability mass outside the support is "moved back" into the domain. Such a reflection could be implemented as follows.

1. Add a `bounds` parameter to the `KernelDensity` constructor with shape `(n_features, 2)` comprising the lower and upper bound in each dimension of the feature space.
2. Raise a `ValueError` in `fit` if one or more samples are outside the specified bounds.
3. Reflect samples to be scored at the boundaries in `score_samples` and combine contributions from all reflections using `logsumexp`.

Alternatively, a new estimator `BoundedKernelDensity` could inherit from `KernelDensity` to keep the `KernelDensity` API from ballooning. Would either be of interest for inclusion in scikit-learn?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Yes, I can have a look. I've got a preliminary implementation [here](https://github.com/tillahoffmann/snippets/blob/ff668a3124bd6b3154d251677ef7af98fd328deb/snippets/stats.py#L73-L197) with an example [here](https://scientific-snippets.readthedocs.io/en/latest/docs/stats.html#snippets.stats.GaussianKernelDensity). If that looks like a reasonable approach, I can adapt the code to better integrate with scikit-learn and make a PR.

Labels: New Feature
Kernel density estimation on (semi-)bounded domains

### Describe the workflow you want to enable

Kernel density estimates for bounded data are biased near the boundary because probability mass "spills out of the domain". It would be great to add a boundary correction to `KernelDensity`. For example, estimating the uniform density on the unit interval is severely biased near the boundaries.

```python
from matplotlib import pyplot as plt
import numpy as np
from sklearn.neighbors import KernelDensity


X = np.random.uniform(0, 1, (10_000, 1))
kde = KernelDensity(bandwidth="silverman").fit(X)
x = np.linspace(0, 1)
plt.plot(x, np.exp(kde.score_samples(x[:, None])))
```

![image](https://github.com/scikit-learn/scikit-learn/assets/966348/dc213fc8-7475-4f82-82e3-d3b883ee44e7)


### Describe your proposed solution

Reflection methods reflect samples to be scored at the boundary such that the probability mass outside the support is "moved back" into the domain. Such a reflection could be implemented as follows.

1. Add a `bounds` parameter to the `KernelDensity` constructor with shape `(n_features, 2)` comprising the lower and upper bound in each dimension of the feature space.
2. Raise a `ValueError` in `fit` if one or more samples are outside the specified bounds.
3. Reflect samples to be scored at the boundaries in `score_samples` and combine contributions from all reflections using `logsumexp`.

Alternatively, a new estimator `BoundedKernelDensity` could inherit from `KernelDensity` to keep the `KernelDensity` API from ballooning. Would either be of interest for inclusion in scikit-learn?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Yep this look and we need additional unit tests and maybe a small example somewhere.

Labels: New Feature
Kernel density estimation on (semi-)bounded domains

### Describe the workflow you want to enable

Kernel density estimates for bounded data are biased near the boundary because probability mass "spills out of the domain". It would be great to add a boundary correction to `KernelDensity`. For example, estimating the uniform density on the unit interval is severely biased near the boundaries.

```python
from matplotlib import pyplot as plt
import numpy as np
from sklearn.neighbors import KernelDensity


X = np.random.uniform(0, 1, (10_000, 1))
kde = KernelDensity(bandwidth="silverman").fit(X)
x = np.linspace(0, 1)
plt.plot(x, np.exp(kde.score_samples(x[:, None])))
```

![image](https://github.com/scikit-learn/scikit-learn/assets/966348/dc213fc8-7475-4f82-82e3-d3b883ee44e7)


### Describe your proposed solution

Reflection methods reflect samples to be scored at the boundary such that the probability mass outside the support is "moved back" into the domain. Such a reflection could be implemented as follows.

1. Add a `bounds` parameter to the `KernelDensity` constructor with shape `(n_features, 2)` comprising the lower and upper bound in each dimension of the feature space.
2. Raise a `ValueError` in `fit` if one or more samples are outside the specified bounds.
3. Reflect samples to be scored at the boundaries in `score_samples` and combine contributions from all reflections using `logsumexp`.

Alternatively, a new estimator `BoundedKernelDensity` could inherit from `KernelDensity` to keep the `KernelDensity` API from ballooning. Would either be of interest for inclusion in scikit-learn?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
A first version of the implementation with tests and an example is available in https://github.com/scikit-learn/scikit-learn/pull/29370.

Labels: New Feature