Talip7/scikit-learn-issues-multilabel huggingface dataset:
HDBSCAN modifies input precomputed distance matrix

### Describe the bug

When using `sklearn.cluster.HDBSCAN` with `metric="precomputed"`, the input distance matrix is modified after calling `fit_predict()`. The original `hdbscan` package (v0.8.40) works correctly.  

### Steps/Code to Reproduce
```py
import numpy as np
from sklearn.cluster import HDBSCAN

rmsd_matrix = np.random.rand(5, 5)
rmsd_matrix = (rmsd_matrix + rmsd_matrix.T) / 2
np.fill_diagonal(rmsd_matrix, 0)

print("Before HDBSCAN:")
print(rmsd_matrix)

hdb = HDBSCAN(metric="precomputed", min_cluster_size=2)
hdb.fit_predict(rmsd_matrix)

print("\nAfter HDBSCAN:")
print(rmsd_matrix)  # Matrix is changed!
```

### Expected Results

Input matrix should remain unchanged (as in original hdbscan).

### Actual Results

Input matrix is changed

### Versions

```shell
System:
    python: 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]
executable: /home/username/project/bin/python3
   machine: Linux-6.14.0-27-generic-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.7.0
          pip: 24.0
   setuptools: 80.9.0
        numpy: 2.2.6
        scipy: 1.16.0
       Cython: None
       pandas: 2.3.0
   matplotlib: 3.10.3
       joblib: 1.5.1
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-56d6093b.so
        version: 0.3.29
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/scipy.libs/libscipy_openblas-68440149.so
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 18
         prefix: libgomp
       filepath: /home/username/project/lib/python3.12/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
@Micky774 @adrinjalali I went through the standards on how scikit-learn implements a deprecation cycle in `contributing.rst` . If we are going forward with a deprecation cycle, we can make default value of `copy="warn"` and raise a warning that the default value of `copy` is going to change from False to True in version x, if user uses the default value of copy. Additionally, making some minor changes to `_parameter_constraints`,etc to maintain consistency for this change.

Does this works?

Labels: Bug, module:cluster
HDBSCAN modifies input precomputed distance matrix

### Describe the bug

When using `sklearn.cluster.HDBSCAN` with `metric="precomputed"`, the input distance matrix is modified after calling `fit_predict()`. The original `hdbscan` package (v0.8.40) works correctly.  

### Steps/Code to Reproduce
```py
import numpy as np
from sklearn.cluster import HDBSCAN

rmsd_matrix = np.random.rand(5, 5)
rmsd_matrix = (rmsd_matrix + rmsd_matrix.T) / 2
np.fill_diagonal(rmsd_matrix, 0)

print("Before HDBSCAN:")
print(rmsd_matrix)

hdb = HDBSCAN(metric="precomputed", min_cluster_size=2)
hdb.fit_predict(rmsd_matrix)

print("\nAfter HDBSCAN:")
print(rmsd_matrix)  # Matrix is changed!
```

### Expected Results

Input matrix should remain unchanged (as in original hdbscan).

### Actual Results

Input matrix is changed

### Versions

```shell
System:
    python: 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]
executable: /home/username/project/bin/python3
   machine: Linux-6.14.0-27-generic-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.7.0
          pip: 24.0
   setuptools: 80.9.0
        numpy: 2.2.6
        scipy: 1.16.0
       Cython: None
       pandas: 2.3.0
   matplotlib: 3.10.3
       joblib: 1.5.1
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-56d6093b.so
        version: 0.3.29
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/scipy.libs/libscipy_openblas-68440149.so
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 18
         prefix: libgomp
       filepath: /home/username/project/lib/python3.12/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
> @Micky774 @adrinjalali I went through the standards on how scikit-learn implements a deprecation cycle in `contributing.rst` . If we are going forward with a deprecation cycle, we can make default value of `copy="warn"` and raise a warning that the default value of `copy` is going to change from False to True in version x, if user uses the default value of copy. Additionally, making some minor changes to `_parameter_constraints`,etc to maintain consistency for this change.
> 
> Does this works?

That sounds good to me. Feel free to ping me for review on that PR

Labels: Bug, module:cluster
HDBSCAN modifies input precomputed distance matrix

### Describe the bug

When using `sklearn.cluster.HDBSCAN` with `metric="precomputed"`, the input distance matrix is modified after calling `fit_predict()`. The original `hdbscan` package (v0.8.40) works correctly.  

### Steps/Code to Reproduce
```py
import numpy as np
from sklearn.cluster import HDBSCAN

rmsd_matrix = np.random.rand(5, 5)
rmsd_matrix = (rmsd_matrix + rmsd_matrix.T) / 2
np.fill_diagonal(rmsd_matrix, 0)

print("Before HDBSCAN:")
print(rmsd_matrix)

hdb = HDBSCAN(metric="precomputed", min_cluster_size=2)
hdb.fit_predict(rmsd_matrix)

print("\nAfter HDBSCAN:")
print(rmsd_matrix)  # Matrix is changed!
```

### Expected Results

Input matrix should remain unchanged (as in original hdbscan).

### Actual Results

Input matrix is changed

### Versions

```shell
System:
    python: 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]
executable: /home/username/project/bin/python3
   machine: Linux-6.14.0-27-generic-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.7.0
          pip: 24.0
   setuptools: 80.9.0
        numpy: 2.2.6
        scipy: 1.16.0
       Cython: None
       pandas: 2.3.0
   matplotlib: 3.10.3
       joblib: 1.5.1
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-56d6093b.so
        version: 0.3.29
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/scipy.libs/libscipy_openblas-68440149.so
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 18
         prefix: libgomp
       filepath: /home/username/project/lib/python3.12/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
Which version should I mention for when this change will go live?

Labels: Bug, module:cluster
HDBSCAN modifies input precomputed distance matrix

### Describe the bug

When using `sklearn.cluster.HDBSCAN` with `metric="precomputed"`, the input distance matrix is modified after calling `fit_predict()`. The original `hdbscan` package (v0.8.40) works correctly.  

### Steps/Code to Reproduce
```py
import numpy as np
from sklearn.cluster import HDBSCAN

rmsd_matrix = np.random.rand(5, 5)
rmsd_matrix = (rmsd_matrix + rmsd_matrix.T) / 2
np.fill_diagonal(rmsd_matrix, 0)

print("Before HDBSCAN:")
print(rmsd_matrix)

hdb = HDBSCAN(metric="precomputed", min_cluster_size=2)
hdb.fit_predict(rmsd_matrix)

print("\nAfter HDBSCAN:")
print(rmsd_matrix)  # Matrix is changed!
```

### Expected Results

Input matrix should remain unchanged (as in original hdbscan).

### Actual Results

Input matrix is changed

### Versions

```shell
System:
    python: 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]
executable: /home/username/project/bin/python3
   machine: Linux-6.14.0-27-generic-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.7.0
          pip: 24.0
   setuptools: 80.9.0
        numpy: 2.2.6
        scipy: 1.16.0
       Cython: None
       pandas: 2.3.0
   matplotlib: 3.10.3
       joblib: 1.5.1
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-56d6093b.so
        version: 0.3.29
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/scipy.libs/libscipy_openblas-68440149.so
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 18
         prefix: libgomp
       filepath: /home/username/project/lib/python3.12/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
That would be 1.10

Labels: Bug, module:cluster
⚠️ CI failed on Linux_Runs.pylatest_conda_forge_mkl (last failure: Aug 17, 2025) ⚠️

**CI is still failing on [Linux_Runs.pylatest_conda_forge_mkl](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=79126&view=logs&j=dde5042c-7464-5d47-9507-31bdd2ee0a3a)** (Aug 17, 2025)
- Test Collection Failure

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=79253&view=logs&j=dde5042c-7464-5d47-9507-31bdd2ee0a3a) on Aug 22, 2025

Labels: Needs Triage
⚠️ CI failed on Linux_Runs.pylatest_conda_forge_mkl (last failure: Aug 17, 2025) ⚠️

**CI is still failing on [Linux_Runs.pylatest_conda_forge_mkl](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=79126&view=logs&j=dde5042c-7464-5d47-9507-31bdd2ee0a3a)** (Aug 17, 2025)
- Test Collection Failure

COMMENT:
Fixed by https://github.com/scikit-learn/scikit-learn/pull/31935

Labels: Needs Triage
QuantileTransformer is incredibly slow

### Describe the workflow you want to enable

This is a feature request to improve performance of the QuantileTransformer. It takes ~60 minutes to fit, uses a huge amount of memory when transforming large non-sparse dataframes with 30M+ rows and 500 columns. It also does not support sample_weight.  Ideally it should be as fast as catboost's Pool quantize method, which does many of the same computations in a fraction of the time:
https://catboost.ai/docs/en/concepts/python-reference_pool_quantized


### Describe your proposed solution

See source code for https://catboost.ai/docs/en/concepts/python-reference_pool_quantized

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Please provide evidence of your claims, best by a **minimal reproducible example**.
To speed up fitting, you can use `subsample`.
Catboost quantize seems to do something different.

Labels: New Feature, Needs Triage
QuantileTransformer is incredibly slow

### Describe the workflow you want to enable

This is a feature request to improve performance of the QuantileTransformer. It takes ~60 minutes to fit, uses a huge amount of memory when transforming large non-sparse dataframes with 30M+ rows and 500 columns. It also does not support sample_weight.  Ideally it should be as fast as catboost's Pool quantize method, which does many of the same computations in a fraction of the time:
https://catboost.ai/docs/en/concepts/python-reference_pool_quantized


### Describe your proposed solution

See source code for https://catboost.ai/docs/en/concepts/python-reference_pool_quantized

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Please do not prematurely close this issue. Here is an example. Aside from being slow, memory spikes during the quantile transformation and it is never released back into memory. The point is for large datasets, quantile transformation takes even longer than model training.

```python

from sklearn.datasets import load_iris
from sklearn import preprocessing
from sklearn.model_selection import train_test_split
import numpy as np
from sklearn.preprocessing import QuantileTransformer
from sklearn.compose import ColumnTransformer
import matplotlib.pyplot as plt
from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from time import time
import psutil, os, gc
import pandas as pd


def current_mem()   :
    process = psutil.Process(os.getpid())
    rssgb = process.memory_info().rss / 2 ** 30
    print(rssgb)
    return rssgb

Xtrain, ytrain = make_regression(n_samples=1000000, n_features=800, noise=0.1, random_state=0)
columns=['C{}'.format(i) for i in range(800)]

dftrain = pd.DataFrame(Xtrain, columns=columns)

st = time()
current_mem()
columns_to_scale = columns[-500:]
gc.collect()
quantile_transformer = ColumnTransformer([('qts', QuantileTransformer(n_quantiles=20, output_distribution="normal", subsample=1e9, copy=False, random_state=0), columns_to_scale)], remainder='passthrough', n_jobs=None, verbose=False, verbose_feature_names_out=False)

dftrain = quantile_transformer.fit_transform(dftrain)
gc.collect()
et = time()
current_mem()
print("Time elapsed:{}".format(et-st))

Output:
10.222393035888672
16.9678955078125
Time elapsed:110.87386012077332


current_mem()
16.9678955078125

```


Comparison with pool quantize method:
```python

from catboost import Pool

st = time()
pool = Pool(data=dftrain, label=None)
pool.quantize(border_count=20, feature_border_type='Median', random_seed=0)
et = time()
print("Time elapsed:{}".format(et-st))

Output:
Time elapsed:5.983074903488159

```

Labels: New Feature, Needs Triage
QuantileTransformer is incredibly slow

### Describe the workflow you want to enable

This is a feature request to improve performance of the QuantileTransformer. It takes ~60 minutes to fit, uses a huge amount of memory when transforming large non-sparse dataframes with 30M+ rows and 500 columns. It also does not support sample_weight.  Ideally it should be as fast as catboost's Pool quantize method, which does many of the same computations in a fraction of the time:
https://catboost.ai/docs/en/concepts/python-reference_pool_quantized


### Describe your proposed solution

See source code for https://catboost.ai/docs/en/concepts/python-reference_pool_quantized

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
The main bottleneck for `fit` is calculating the `n_quantiles` quantiles for all the columns/features in https://github.com/scikit-learn/scikit-learn/blob/78301f59bcd35c542e9bdabeab04f39e9ca099c3/sklearn/preprocessing/_data.py#L2804
We want to rely on numpy. So there is nothing we can do about it. If you want to improve it, you can contribute to numpy or use `subsample << X.shape[0]`.

For `transform`, I am not 100% sure where the bottleneck is exactly, maybe in calling `scipy.stats.norm.ppf` on every column of X.

About the memory, at your first call to current memory, Xtrain and dftrain are the largest object. Then you call the garbage collector, but it will not remove Xtrain nor (the new) dftrain. The old dftrain might get removed. It very much depends on gc not on scikit-learn.
Bottom line: I don't see any way to change that.

About catboost Pool, I am not convinced **at all** that it is doing the same thing.

Labels: New Feature, Needs Triage
QuantileTransformer is incredibly slow

### Describe the workflow you want to enable

This is a feature request to improve performance of the QuantileTransformer. It takes ~60 minutes to fit, uses a huge amount of memory when transforming large non-sparse dataframes with 30M+ rows and 500 columns. It also does not support sample_weight.  Ideally it should be as fast as catboost's Pool quantize method, which does many of the same computations in a fraction of the time:
https://catboost.ai/docs/en/concepts/python-reference_pool_quantized


### Describe your proposed solution

See source code for https://catboost.ai/docs/en/concepts/python-reference_pool_quantized

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@lorentzenchr  Can you at least allow the user to pass in a list of their own computed quantiles, and bypass this line?
```python
 self.quantiles_ = np.nanpercentile(X, references, axis=0) 
```
i.e, let the user call an instantiate and transform method with kwarg quantiles = array of feature quantiles, which should at least cut the processing time in half.

Labels: New Feature, Needs Triage
QuantileTransformer is incredibly slow

### Describe the workflow you want to enable

This is a feature request to improve performance of the QuantileTransformer. It takes ~60 minutes to fit, uses a huge amount of memory when transforming large non-sparse dataframes with 30M+ rows and 500 columns. It also does not support sample_weight.  Ideally it should be as fast as catboost's Pool quantize method, which does many of the same computations in a fraction of the time:
https://catboost.ai/docs/en/concepts/python-reference_pool_quantized


### Describe your proposed solution

See source code for https://catboost.ai/docs/en/concepts/python-reference_pool_quantized

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
You can use `FunctionTransformer` or write your own estimator.

Labels: New Feature, Needs Triage
QuantileTransformer is incredibly slow

### Describe the workflow you want to enable

This is a feature request to improve performance of the QuantileTransformer. It takes ~60 minutes to fit, uses a huge amount of memory when transforming large non-sparse dataframes with 30M+ rows and 500 columns. It also does not support sample_weight.  Ideally it should be as fast as catboost's Pool quantize method, which does many of the same computations in a fraction of the time:
https://catboost.ai/docs/en/concepts/python-reference_pool_quantized


### Describe your proposed solution

See source code for https://catboost.ai/docs/en/concepts/python-reference_pool_quantized

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
If you insist on using numpy in the fit method, you should at least compile with numba or assert that there are no nans because np.nanpercentile is notoriously slow. See [this thread](https://stackoverflow.com/questions/60015245/numpy-nanpercentile-is-extremely-slow)
```python

@numba.jit(nopython=True, cache=True)
def nan_percentile_axis0(arr, percentiles):
    """Faster implementation of np.nanpercentile
    
    This implementation always takes the percentile along axis 0.
    Uses numba to speed up the calculation by more than 7x.

    Function is equivalent to np.nanpercentile(arr, <percentiles>, axis=0)

    Params:
        arr (np.array): Array to calculate percentiles for
        percentiles (np.array): 1D array of percentiles to calculate

    Returns:
        (np.array) Array with first dimension corresponding to
            values as passed in percentiles

    """
    shape = arr.shape
    arr = arr.reshape((arr.shape[0], -1))
    out = np.empty((len(percentiles), arr.shape[1]))
    for i in range(arr.shape[1]):
        out[:,i] = np.nanpercentile(arr[:,i], percentiles)
    shape = (out.shape[0], *shape[1:])
    return out.reshape(shape)
```

Labels: New Feature, Needs Triage
QuantileTransformer is incredibly slow

### Describe the workflow you want to enable

This is a feature request to improve performance of the QuantileTransformer. It takes ~60 minutes to fit, uses a huge amount of memory when transforming large non-sparse dataframes with 30M+ rows and 500 columns. It also does not support sample_weight.  Ideally it should be as fast as catboost's Pool quantize method, which does many of the same computations in a fraction of the time:
https://catboost.ai/docs/en/concepts/python-reference_pool_quantized


### Describe your proposed solution

See source code for https://catboost.ai/docs/en/concepts/python-reference_pool_quantized

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I don't insist. We, the scikit-learn maintainers are very conservative with dependencies. Numba is a very heavy (and tricky) dependency that has been discussed in the past and discarded.

With numpy 2.1.3, I don't measure a significant difference between np.percentile and np.nanpercentile with random X, `X.shape = 1000_000, 100` and `references_ = np.linspace(0, 1, 1000, endpoint=True)`:
```
%time np.percentile(X, references_, axis=0)
CPU times: user 4.51 s, sys: 239 ms, total: 4.75 s
Wall time: 4.75 s

%time np.nanpercentile(X, references_, axis=0)
CPU times: user 4.44 s, sys: 10.5 ms, total: 4.45 s
Wall time: 4.46 s

Labels: New Feature, Needs Triage
QuantileTransformer is incredibly slow

### Describe the workflow you want to enable

This is a feature request to improve performance of the QuantileTransformer. It takes ~60 minutes to fit, uses a huge amount of memory when transforming large non-sparse dataframes with 30M+ rows and 500 columns. It also does not support sample_weight.  Ideally it should be as fast as catboost's Pool quantize method, which does many of the same computations in a fraction of the time:
https://catboost.ai/docs/en/concepts/python-reference_pool_quantized


### Describe your proposed solution

See source code for https://catboost.ai/docs/en/concepts/python-reference_pool_quantized

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
What about supporting sample_weight with the transformer as is the norm with other sklearn transformers? Or is that also out of scope

Labels: New Feature, Needs Triage