Talip7/scikit-learn-issues-multilabel huggingface dataset:
Stable extender contract via `fit` / `_fit` resp `predict` / `_predict` separation

### Describe the workflow you want to enable

tl;dr, I am suggestion to refactor `scikit-learn` internals to a layer separation with boilerplate between `fit` and `_fit` resp `predict` and `_predict` methods, to make extender interfaces more stable. Also see https://github.com/scikit-learn/scikit-learn/issues/31728

More background: Currently, every time `scikit-learn` releases a new minor version - e.g., 1.5.0, 1.6.0, 1.7.0 - compliant extensions, e.g., custom transformers, classifiers, etc, break - specifically, referring to the API conformance as tested through `check_estimator` or `parametrize_with_checks`.

These repeated breakages in the "extender contract" contrast the stability of the usage contract, which is stable and professionally managed.

For a package like `scikit-learn` which means to be a standard not just for ML algorithms but also an API standard that everyone uses, this is not a good state to be in - "do not break user code" is the maxim that gets broken for power users writing extensions.

Of course maintaining downwards compatibility is not always possible, but nothing should break without a proper warning.

### Describe your proposed solution

The `fit`/`_fit` separation would ensure stability of the extension contract - and would also allow to build secondary deprecation patterns in relation to it.

The (oop) pattern this would implement is the so-called "template pattern".

It would allow to remove likely changing parts such as the boilerplate (e.g., `validate_data` vs `_validate_data` and such) from the extension locus, and thus completely prevent breakage in relation to boilerplate changes.
Reference: https://refactoring.guru/design-patterns/template-method

Examples of how this can be used to improve stability:

* `sktime`, for a different API, has a separation between `fit` calling an internal `_fit`, where change-prone boilerplate is sandwiched between a stable user contract (`fit`) and a stable extender contract (`_fit`); similarly `predict` and `_predict`
* `feature-engine` overrides the `BaseTransformer` `scikit-learn` extension contract with a similar pattern using `super()` calls in `fit` etc.

The proposed `fit`/`_fit` pairing that combines strategy and template pattern can be introduced easily via pure internal refactoring - this also add a layer of tests that checks the stability of the extension contract.

As a side effect, this would likely reduce the amount of copy-pasted boilerplate among the core estimators in `scikit-learn` proper as well.

### Describe alternatives you've considered, if relevant

The obvious alternative is "do nothing" - which will lead to packages with extensions producing their own version of mitigating patterns for the unstable status quo.

The problem with this is that the extending packages are mutually incompatible with their extender pattern. It would be better if there is a single stable extension pattern prescribed - or at least recommended - by `scikit-learn`.

### Additional context

_No response_

COMMENT:
so... what happens now? Anything I should be doing?

Labels: RFC, Developer API
HDBSCAN modifies input precomputed distance matrix

### Describe the bug

When using `sklearn.cluster.HDBSCAN` with `metric="precomputed"`, the input distance matrix is modified after calling `fit_predict()`. The original `hdbscan` package (v0.8.40) works correctly.  

### Steps/Code to Reproduce
```py
import numpy as np
from sklearn.cluster import HDBSCAN

rmsd_matrix = np.random.rand(5, 5)
rmsd_matrix = (rmsd_matrix + rmsd_matrix.T) / 2
np.fill_diagonal(rmsd_matrix, 0)

print("Before HDBSCAN:")
print(rmsd_matrix)

hdb = HDBSCAN(metric="precomputed", min_cluster_size=2)
hdb.fit_predict(rmsd_matrix)

print("\nAfter HDBSCAN:")
print(rmsd_matrix)  # Matrix is changed!
```

### Expected Results

Input matrix should remain unchanged (as in original hdbscan).

### Actual Results

Input matrix is changed

### Versions

```shell
System:
    python: 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]
executable: /home/username/project/bin/python3
   machine: Linux-6.14.0-27-generic-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.7.0
          pip: 24.0
   setuptools: 80.9.0
        numpy: 2.2.6
        scipy: 1.16.0
       Cython: None
       pandas: 2.3.0
   matplotlib: 3.10.3
       joblib: 1.5.1
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-56d6093b.so
        version: 0.3.29
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/scipy.libs/libscipy_openblas-68440149.so
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 18
         prefix: libgomp
       filepath: /home/username/project/lib/python3.12/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
I would like to contribute

Labels: Bug, module:cluster
HDBSCAN modifies input precomputed distance matrix

### Describe the bug

When using `sklearn.cluster.HDBSCAN` with `metric="precomputed"`, the input distance matrix is modified after calling `fit_predict()`. The original `hdbscan` package (v0.8.40) works correctly.  

### Steps/Code to Reproduce
```py
import numpy as np
from sklearn.cluster import HDBSCAN

rmsd_matrix = np.random.rand(5, 5)
rmsd_matrix = (rmsd_matrix + rmsd_matrix.T) / 2
np.fill_diagonal(rmsd_matrix, 0)

print("Before HDBSCAN:")
print(rmsd_matrix)

hdb = HDBSCAN(metric="precomputed", min_cluster_size=2)
hdb.fit_predict(rmsd_matrix)

print("\nAfter HDBSCAN:")
print(rmsd_matrix)  # Matrix is changed!
```

### Expected Results

Input matrix should remain unchanged (as in original hdbscan).

### Actual Results

Input matrix is changed

### Versions

```shell
System:
    python: 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]
executable: /home/username/project/bin/python3
   machine: Linux-6.14.0-27-generic-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.7.0
          pip: 24.0
   setuptools: 80.9.0
        numpy: 2.2.6
        scipy: 1.16.0
       Cython: None
       pandas: 2.3.0
   matplotlib: 3.10.3
       joblib: 1.5.1
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-56d6093b.so
        version: 0.3.29
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/scipy.libs/libscipy_openblas-68440149.so
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 18
         prefix: libgomp
       filepath: /home/username/project/lib/python3.12/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
I’m currently working on this issue.
I have confirmed that when passing a dense precomputed distance matrix to HDBSCAN, scikit-learn’s implementation mutates the matrix in place. This should not happen as the input pairwise distance matrix should remain unchanged after fitting.
Earlier, I suspected that it was due to `force_writeable=True` but maybe that's not the case. I'm currently investigating the issue further.

Labels: Bug, module:cluster
HDBSCAN modifies input precomputed distance matrix

### Describe the bug

When using `sklearn.cluster.HDBSCAN` with `metric="precomputed"`, the input distance matrix is modified after calling `fit_predict()`. The original `hdbscan` package (v0.8.40) works correctly.  

### Steps/Code to Reproduce
```py
import numpy as np
from sklearn.cluster import HDBSCAN

rmsd_matrix = np.random.rand(5, 5)
rmsd_matrix = (rmsd_matrix + rmsd_matrix.T) / 2
np.fill_diagonal(rmsd_matrix, 0)

print("Before HDBSCAN:")
print(rmsd_matrix)

hdb = HDBSCAN(metric="precomputed", min_cluster_size=2)
hdb.fit_predict(rmsd_matrix)

print("\nAfter HDBSCAN:")
print(rmsd_matrix)  # Matrix is changed!
```

### Expected Results

Input matrix should remain unchanged (as in original hdbscan).

### Actual Results

Input matrix is changed

### Versions

```shell
System:
    python: 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]
executable: /home/username/project/bin/python3
   machine: Linux-6.14.0-27-generic-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.7.0
          pip: 24.0
   setuptools: 80.9.0
        numpy: 2.2.6
        scipy: 1.16.0
       Cython: None
       pandas: 2.3.0
   matplotlib: 3.10.3
       joblib: 1.5.1
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-56d6093b.so
        version: 0.3.29
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/scipy.libs/libscipy_openblas-68440149.so
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 18
         prefix: libgomp
       filepath: /home/username/project/lib/python3.12/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
HDBSCAN(metric="precomputed") mutates the input distance matrix in-place, unlike the original hdbscan library.
Add X = X.copy() when metric == "precomputed" to prevent mutation.

Labels: Bug, module:cluster
HDBSCAN modifies input precomputed distance matrix

### Describe the bug

When using `sklearn.cluster.HDBSCAN` with `metric="precomputed"`, the input distance matrix is modified after calling `fit_predict()`. The original `hdbscan` package (v0.8.40) works correctly.  

### Steps/Code to Reproduce
```py
import numpy as np
from sklearn.cluster import HDBSCAN

rmsd_matrix = np.random.rand(5, 5)
rmsd_matrix = (rmsd_matrix + rmsd_matrix.T) / 2
np.fill_diagonal(rmsd_matrix, 0)

print("Before HDBSCAN:")
print(rmsd_matrix)

hdb = HDBSCAN(metric="precomputed", min_cluster_size=2)
hdb.fit_predict(rmsd_matrix)

print("\nAfter HDBSCAN:")
print(rmsd_matrix)  # Matrix is changed!
```

### Expected Results

Input matrix should remain unchanged (as in original hdbscan).

### Actual Results

Input matrix is changed

### Versions

```shell
System:
    python: 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]
executable: /home/username/project/bin/python3
   machine: Linux-6.14.0-27-generic-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.7.0
          pip: 24.0
   setuptools: 80.9.0
        numpy: 2.2.6
        scipy: 1.16.0
       Cython: None
       pandas: 2.3.0
   matplotlib: 3.10.3
       joblib: 1.5.1
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-56d6093b.so
        version: 0.3.29
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/scipy.libs/libscipy_openblas-68440149.so
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 18
         prefix: libgomp
       filepath: /home/username/project/lib/python3.12/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
@Sarthakpurii now that you're investigating, could you please have a look at other estimators where we have a `copy` arg and see how the default behavior there is, and if there's any other case of precomputed input which could be a similar case. I'd be happy to have this fixed if it's not consistent with other places.

Labels: Bug, module:cluster
HDBSCAN modifies input precomputed distance matrix

### Describe the bug

When using `sklearn.cluster.HDBSCAN` with `metric="precomputed"`, the input distance matrix is modified after calling `fit_predict()`. The original `hdbscan` package (v0.8.40) works correctly.  

### Steps/Code to Reproduce
```py
import numpy as np
from sklearn.cluster import HDBSCAN

rmsd_matrix = np.random.rand(5, 5)
rmsd_matrix = (rmsd_matrix + rmsd_matrix.T) / 2
np.fill_diagonal(rmsd_matrix, 0)

print("Before HDBSCAN:")
print(rmsd_matrix)

hdb = HDBSCAN(metric="precomputed", min_cluster_size=2)
hdb.fit_predict(rmsd_matrix)

print("\nAfter HDBSCAN:")
print(rmsd_matrix)  # Matrix is changed!
```

### Expected Results

Input matrix should remain unchanged (as in original hdbscan).

### Actual Results

Input matrix is changed

### Versions

```shell
System:
    python: 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]
executable: /home/username/project/bin/python3
   machine: Linux-6.14.0-27-generic-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.7.0
          pip: 24.0
   setuptools: 80.9.0
        numpy: 2.2.6
        scipy: 1.16.0
       Cython: None
       pandas: 2.3.0
   matplotlib: 3.10.3
       joblib: 1.5.1
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-56d6093b.so
        version: 0.3.29
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/scipy.libs/libscipy_openblas-68440149.so
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 18
         prefix: libgomp
       filepath: /home/username/project/lib/python3.12/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
@adrinjalali I investigated other estimators with the `copy` parameter. Every single one defaults to `copy=True`. Here is the list of all the estimators for your convenience to quickly go through - `FactorAnalysis`, `PLSSVD`, `_PLS`, `AffinityPropagation`, `IncrementalPCA`, `PCA`, `SimpleImputer`, `OrthogonalMatchingPursuitCV`, `KNNImputer` & all normalizers/scalers. `HDBSCAN` is the only estimator having `copy=False`.

Earlier, I was hesitant to set HDBSCAN's default `copy=True` because I was concerned about the performance and memory overhead and scikit-learn's broader design patterns. This is the reason I handled the `copy` to be `True` just for dense precomputed distance metrics in my recent PR, but now I feel setting default `copy=True` would be the better choice to maintain consistency. I would love to raise another PR correcting the same if you allow.

What are your thoughts on this?

Labels: Bug, module:cluster
HDBSCAN modifies input precomputed distance matrix

### Describe the bug

When using `sklearn.cluster.HDBSCAN` with `metric="precomputed"`, the input distance matrix is modified after calling `fit_predict()`. The original `hdbscan` package (v0.8.40) works correctly.  

### Steps/Code to Reproduce
```py
import numpy as np
from sklearn.cluster import HDBSCAN

rmsd_matrix = np.random.rand(5, 5)
rmsd_matrix = (rmsd_matrix + rmsd_matrix.T) / 2
np.fill_diagonal(rmsd_matrix, 0)

print("Before HDBSCAN:")
print(rmsd_matrix)

hdb = HDBSCAN(metric="precomputed", min_cluster_size=2)
hdb.fit_predict(rmsd_matrix)

print("\nAfter HDBSCAN:")
print(rmsd_matrix)  # Matrix is changed!
```

### Expected Results

Input matrix should remain unchanged (as in original hdbscan).

### Actual Results

Input matrix is changed

### Versions

```shell
System:
    python: 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]
executable: /home/username/project/bin/python3
   machine: Linux-6.14.0-27-generic-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.7.0
          pip: 24.0
   setuptools: 80.9.0
        numpy: 2.2.6
        scipy: 1.16.0
       Cython: None
       pandas: 2.3.0
   matplotlib: 3.10.3
       joblib: 1.5.1
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-56d6093b.so
        version: 0.3.29
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/scipy.libs/libscipy_openblas-68440149.so
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 18
         prefix: libgomp
       filepath: /home/username/project/lib/python3.12/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
@adrinjalali I am waiting for your views on this.

Labels: Bug, module:cluster
HDBSCAN modifies input precomputed distance matrix

### Describe the bug

When using `sklearn.cluster.HDBSCAN` with `metric="precomputed"`, the input distance matrix is modified after calling `fit_predict()`. The original `hdbscan` package (v0.8.40) works correctly.  

### Steps/Code to Reproduce
```py
import numpy as np
from sklearn.cluster import HDBSCAN

rmsd_matrix = np.random.rand(5, 5)
rmsd_matrix = (rmsd_matrix + rmsd_matrix.T) / 2
np.fill_diagonal(rmsd_matrix, 0)

print("Before HDBSCAN:")
print(rmsd_matrix)

hdb = HDBSCAN(metric="precomputed", min_cluster_size=2)
hdb.fit_predict(rmsd_matrix)

print("\nAfter HDBSCAN:")
print(rmsd_matrix)  # Matrix is changed!
```

### Expected Results

Input matrix should remain unchanged (as in original hdbscan).

### Actual Results

Input matrix is changed

### Versions

```shell
System:
    python: 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]
executable: /home/username/project/bin/python3
   machine: Linux-6.14.0-27-generic-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.7.0
          pip: 24.0
   setuptools: 80.9.0
        numpy: 2.2.6
        scipy: 1.16.0
       Cython: None
       pandas: 2.3.0
   matplotlib: 3.10.3
       joblib: 1.5.1
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-56d6093b.so
        version: 0.3.29
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/scipy.libs/libscipy_openblas-68440149.so
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 18
         prefix: libgomp
       filepath: /home/username/project/lib/python3.12/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
Then I think we can have a PR changing the default value, w/o deprecation cycle, and have it as a FIX. I wonder what @Micky774 thinks about it though.

Labels: Bug, module:cluster
HDBSCAN modifies input precomputed distance matrix

### Describe the bug

When using `sklearn.cluster.HDBSCAN` with `metric="precomputed"`, the input distance matrix is modified after calling `fit_predict()`. The original `hdbscan` package (v0.8.40) works correctly.  

### Steps/Code to Reproduce
```py
import numpy as np
from sklearn.cluster import HDBSCAN

rmsd_matrix = np.random.rand(5, 5)
rmsd_matrix = (rmsd_matrix + rmsd_matrix.T) / 2
np.fill_diagonal(rmsd_matrix, 0)

print("Before HDBSCAN:")
print(rmsd_matrix)

hdb = HDBSCAN(metric="precomputed", min_cluster_size=2)
hdb.fit_predict(rmsd_matrix)

print("\nAfter HDBSCAN:")
print(rmsd_matrix)  # Matrix is changed!
```

### Expected Results

Input matrix should remain unchanged (as in original hdbscan).

### Actual Results

Input matrix is changed

### Versions

```shell
System:
    python: 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]
executable: /home/username/project/bin/python3
   machine: Linux-6.14.0-27-generic-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.7.0
          pip: 24.0
   setuptools: 80.9.0
        numpy: 2.2.6
        scipy: 1.16.0
       Cython: None
       pandas: 2.3.0
   matplotlib: 3.10.3
       joblib: 1.5.1
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-56d6093b.so
        version: 0.3.29
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 18
         prefix: libscipy_openblas
       filepath: /home/username/project/lib/python3.12/site-packages/scipy.libs/libscipy_openblas-68440149.so
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 18
         prefix: libgomp
       filepath: /home/username/project/lib/python3.12/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
> Then I think we can have a PR changing the default value, w/o deprecation cycle, and have it as a FIX. I wonder what [@Micky774](https://github.com/Micky774) thinks about it though.

It's worth making it consistent with other estimators if it is currently off. I think it might be worth a deprecation cycle though, since it could meaningfully break some workflows that perhaps already assumed that a copy was being made, and hence wouldn't immediately know how to correct for the change. Thank you for bringing this to our attention @Sarthakpurii and @tolyanchik5 :)

Labels: Bug, module:cluster