Talip7/scikit-learn-issues-multilabel huggingface dataset:
Clarify the differences between `LinearSVC` and `SVC`

### Describe the issue linked to the documentation

Currently the documentation for `LinearSVC` says that is "Similar to SVC with parameter `kernel="linear"`, but implemented in terms of liblinear rather than libsvm, so it has more flexibility in the choice of penalties and loss functions and should scale better to large numbers of samples".

This is confusing, because by default they are not equivalent. `LinearSVC` has a different default loss function and regularizes the intercept, as mentioned in these StackOverflow posts:
- https://stackoverflow.com/questions/45384185/what-is-the-difference-between-linearsvc-and-svckernel-linear
- https://stackoverflow.com/questions/35076586/when-should-one-use-linearsvc-or-svc
- https://stackoverflow.com/questions/33843981/under-what-parameters-are-svc-and-linearsvc-in-scikit-learn-equivalent

As a result, the current documentation can mislead and give result to research that does the wrong thing, as it almost happened to me.

### Suggest a potential alternative/fix

A big red box with a warning should be at the beginning of the documentation for `LinearSVC`, describing the ways in which differs from `SVC` and that it is not a "true" linear SVM (because of the intercept regularization).

In the future, it would be desirable in my opinion to change the default values/implementation of `LinearSVC` so that both `LinearSVC` and `SVC` with parameter `kernel="linear"` solve the same optimization problem by default. Otherwise the risk of it leading to wrong research results is very big.

COMMENT:
It seems that scikit-learn uses an old version of liblinear. At least, in the current liblinear version an attribute `regularize_bias` seems to be present in the parameter struct, allowing to have an unregularized intercept, if I am not mistaken.

Labels: Documentation
Clarify the differences between `LinearSVC` and `SVC`

### Describe the issue linked to the documentation

Currently the documentation for `LinearSVC` says that is "Similar to SVC with parameter `kernel="linear"`, but implemented in terms of liblinear rather than libsvm, so it has more flexibility in the choice of penalties and loss functions and should scale better to large numbers of samples".

This is confusing, because by default they are not equivalent. `LinearSVC` has a different default loss function and regularizes the intercept, as mentioned in these StackOverflow posts:
- https://stackoverflow.com/questions/45384185/what-is-the-difference-between-linearsvc-and-svckernel-linear
- https://stackoverflow.com/questions/35076586/when-should-one-use-linearsvc-or-svc
- https://stackoverflow.com/questions/33843981/under-what-parameters-are-svc-and-linearsvc-in-scikit-learn-equivalent

As a result, the current documentation can mislead and give result to research that does the wrong thing, as it almost happened to me.

### Suggest a potential alternative/fix

A big red box with a warning should be at the beginning of the documentation for `LinearSVC`, describing the ways in which differs from `SVC` and that it is not a "true" linear SVM (because of the intercept regularization).

In the future, it would be desirable in my opinion to change the default values/implementation of `LinearSVC` so that both `LinearSVC` and `SVC` with parameter `kernel="linear"` solve the same optimization problem by default. Otherwise the risk of it leading to wrong research results is very big.

COMMENT:
with both liblinear and libsvm we've diverged a bit from the original vendored version. If you feel like sending a PR to add this to what we have, it'd be nice @vnmabus

Labels: Documentation
Clarify the differences between `LinearSVC` and `SVC`

### Describe the issue linked to the documentation

Currently the documentation for `LinearSVC` says that is "Similar to SVC with parameter `kernel="linear"`, but implemented in terms of liblinear rather than libsvm, so it has more flexibility in the choice of penalties and loss functions and should scale better to large numbers of samples".

This is confusing, because by default they are not equivalent. `LinearSVC` has a different default loss function and regularizes the intercept, as mentioned in these StackOverflow posts:
- https://stackoverflow.com/questions/45384185/what-is-the-difference-between-linearsvc-and-svckernel-linear
- https://stackoverflow.com/questions/35076586/when-should-one-use-linearsvc-or-svc
- https://stackoverflow.com/questions/33843981/under-what-parameters-are-svc-and-linearsvc-in-scikit-learn-equivalent

As a result, the current documentation can mislead and give result to research that does the wrong thing, as it almost happened to me.

### Suggest a potential alternative/fix

A big red box with a warning should be at the beginning of the documentation for `LinearSVC`, describing the ways in which differs from `SVC` and that it is not a "true" linear SVM (because of the intercept regularization).

In the future, it would be desirable in my opinion to change the default values/implementation of `LinearSVC` so that both `LinearSVC` and `SVC` with parameter `kernel="linear"` solve the same optimization problem by default. Otherwise the risk of it leading to wrong research results is very big.

COMMENT:
> with both liblinear and libsvm we've diverged a bit from the original vendored version. If you feel like sending a PR to add this to what we have, it'd be nice @vnmabus

I do not feel comfortable enough with either liblinear or scikit-learn to make a sucessful PR touching this part. Maybe this should be converted to a different issue in case someone wants to tackle it?

Labels: Documentation
Speed up classification_report

### Describe the workflow you want to enable

I'm concerned with slow execution speed of the classification_report  procedure which makes it barely suitable for production-grade workloads.
On a 8M sample it already takes 15 seconds whereas simple POC numba implementation takes 20 to 40 **milli**seconds. I understand that sklearn code is well covered with tests, has wide functionality, follows style guildlines and best practices, but there should be no excuse for the performace leap with simple POC of magnitude that big (x1000).

```python
import numpy as np
from sklearn.metrics import classification_report

y_true = np.random.randint(0, 2, size=2 ** 23)
y_pred = y_true.copy()
np.random.shuffle(y_pred[2 ** 20 : 2 ** 21])

print(classification_report(y_true=y_true, y_pred=y_pred, digits=10, output_dict=False, zero_division=0,))
```

>               precision    recall  f1-score   support
> 
>            0  0.9373906697 0.9373906697 0.9373906697   4192570
>            1  0.9374424159 0.9374424159 0.9374424159   4196038
> 
>     accuracy                      0.9374165535   8388608
>     macro avg  0.9374165428 0.9374165428 0.9374165428   8388608
>     weighted avg  0.9374165535 0.9374165535 0.9374165535   8388608
> 
> time: 18.6 s (started: 2023-07-08 16:10:08 +03:00)
> 


`print(own_classification_report(y_true=y_true, y_pred=y_pred, zero_division=0))`


> (3930076, 3933544, 262494, 262494, 0.9374165534973145, array([4192570, 4196038], dtype=int64), array([0.93739067, 0.93744242]), array([0.93739067, 0.93744242]), array([0.93739067, 0.93744242]))
> time: 16 ms (started: 2023-07-08 16:11:18 +03:00)

### Describe your proposed solution

```python
import numpy as np
from numba import njit


@njit()
def own_classification_report(y_true: np.ndarray, y_pred: np.ndarray, nclasses: int = 2, zero_division: int = 0):

    correct = np.zeros(nclasses, dtype=np.int64)
    wrong = np.zeros(nclasses, dtype=np.int64)
    for truth, pred in zip(y_true, y_pred):
        if pred == truth:
            correct[pred] += 1
        else:
            wrong[pred] += 1
    tn, tp = correct
    fn, fp = wrong
    accuracy = (tn + tp) / len(y_true)

    groups = np.array([(tn + fn), (tp + fp)])
    supports = np.array([(tn + fp), (tp + fn)])

    precisions = np.array([tn, tp]) / groups
    recalls = np.array([tn, tp]) / supports
    f1s = 2 * (precisions * recalls) / (precisions + recalls)

    for arr in (precisions, recalls, f1s):
        np.nan_to_num(arr, copy=False, nan=zero_division)

    return tn, tp, fn, fp, accuracy, supports, precisions, recalls, f1s
```

### Describe alternatives you've considered, if relevant

Can someone from original classification_report authors take a look please why it's taking so long?

### Additional context

I have the latest versions of scikit-learn and numba to this date, an Intel CPU with AVX.

COMMENT:
Could you please do a profiling and show where in the `classification_report` most of the time is spent? Then we can figure out what the best way improving it would be.

Labels: Performance, Needs Investigation
Speed up classification_report

### Describe the workflow you want to enable

I'm concerned with slow execution speed of the classification_report  procedure which makes it barely suitable for production-grade workloads.
On a 8M sample it already takes 15 seconds whereas simple POC numba implementation takes 20 to 40 **milli**seconds. I understand that sklearn code is well covered with tests, has wide functionality, follows style guildlines and best practices, but there should be no excuse for the performace leap with simple POC of magnitude that big (x1000).

```python
import numpy as np
from sklearn.metrics import classification_report

y_true = np.random.randint(0, 2, size=2 ** 23)
y_pred = y_true.copy()
np.random.shuffle(y_pred[2 ** 20 : 2 ** 21])

print(classification_report(y_true=y_true, y_pred=y_pred, digits=10, output_dict=False, zero_division=0,))
```

>               precision    recall  f1-score   support
> 
>            0  0.9373906697 0.9373906697 0.9373906697   4192570
>            1  0.9374424159 0.9374424159 0.9374424159   4196038
> 
>     accuracy                      0.9374165535   8388608
>     macro avg  0.9374165428 0.9374165428 0.9374165428   8388608
>     weighted avg  0.9374165535 0.9374165535 0.9374165535   8388608
> 
> time: 18.6 s (started: 2023-07-08 16:10:08 +03:00)
> 


`print(own_classification_report(y_true=y_true, y_pred=y_pred, zero_division=0))`


> (3930076, 3933544, 262494, 262494, 0.9374165534973145, array([4192570, 4196038], dtype=int64), array([0.93739067, 0.93744242]), array([0.93739067, 0.93744242]), array([0.93739067, 0.93744242]))
> time: 16 ms (started: 2023-07-08 16:11:18 +03:00)

### Describe your proposed solution

```python
import numpy as np
from numba import njit


@njit()
def own_classification_report(y_true: np.ndarray, y_pred: np.ndarray, nclasses: int = 2, zero_division: int = 0):

    correct = np.zeros(nclasses, dtype=np.int64)
    wrong = np.zeros(nclasses, dtype=np.int64)
    for truth, pred in zip(y_true, y_pred):
        if pred == truth:
            correct[pred] += 1
        else:
            wrong[pred] += 1
    tn, tp = correct
    fn, fp = wrong
    accuracy = (tn + tp) / len(y_true)

    groups = np.array([(tn + fn), (tp + fp)])
    supports = np.array([(tn + fp), (tp + fn)])

    precisions = np.array([tn, tp]) / groups
    recalls = np.array([tn, tp]) / supports
    f1s = 2 * (precisions * recalls) / (precisions + recalls)

    for arr in (precisions, recalls, f1s):
        np.nan_to_num(arr, copy=False, nan=zero_division)

    return tn, tp, fn, fp, accuracy, supports, precisions, recalls, f1s
```

### Describe alternatives you've considered, if relevant

Can someone from original classification_report authors take a look please why it's taking so long?

### Additional context

I have the latest versions of scikit-learn and numba to this date, an Intel CPU with AVX.

COMMENT:
Agreed, adding a keyword to disable sorting to `numpy.unique` & co would make sense to me.

Labels: Performance, Needs Investigation
Speed up classification_report

### Describe the workflow you want to enable

I'm concerned with slow execution speed of the classification_report  procedure which makes it barely suitable for production-grade workloads.
On a 8M sample it already takes 15 seconds whereas simple POC numba implementation takes 20 to 40 **milli**seconds. I understand that sklearn code is well covered with tests, has wide functionality, follows style guildlines and best practices, but there should be no excuse for the performace leap with simple POC of magnitude that big (x1000).

```python
import numpy as np
from sklearn.metrics import classification_report

y_true = np.random.randint(0, 2, size=2 ** 23)
y_pred = y_true.copy()
np.random.shuffle(y_pred[2 ** 20 : 2 ** 21])

print(classification_report(y_true=y_true, y_pred=y_pred, digits=10, output_dict=False, zero_division=0,))
```

>               precision    recall  f1-score   support
> 
>            0  0.9373906697 0.9373906697 0.9373906697   4192570
>            1  0.9374424159 0.9374424159 0.9374424159   4196038
> 
>     accuracy                      0.9374165535   8388608
>     macro avg  0.9374165428 0.9374165428 0.9374165428   8388608
>     weighted avg  0.9374165535 0.9374165535 0.9374165535   8388608
> 
> time: 18.6 s (started: 2023-07-08 16:10:08 +03:00)
> 


`print(own_classification_report(y_true=y_true, y_pred=y_pred, zero_division=0))`


> (3930076, 3933544, 262494, 262494, 0.9374165534973145, array([4192570, 4196038], dtype=int64), array([0.93739067, 0.93744242]), array([0.93739067, 0.93744242]), array([0.93739067, 0.93744242]))
> time: 16 ms (started: 2023-07-08 16:11:18 +03:00)

### Describe your proposed solution

```python
import numpy as np
from numba import njit


@njit()
def own_classification_report(y_true: np.ndarray, y_pred: np.ndarray, nclasses: int = 2, zero_division: int = 0):

    correct = np.zeros(nclasses, dtype=np.int64)
    wrong = np.zeros(nclasses, dtype=np.int64)
    for truth, pred in zip(y_true, y_pred):
        if pred == truth:
            correct[pred] += 1
        else:
            wrong[pred] += 1
    tn, tp = correct
    fn, fp = wrong
    accuracy = (tn + tp) / len(y_true)

    groups = np.array([(tn + fn), (tp + fp)])
    supports = np.array([(tn + fp), (tp + fn)])

    precisions = np.array([tn, tp]) / groups
    recalls = np.array([tn, tp]) / supports
    f1s = 2 * (precisions * recalls) / (precisions + recalls)

    for arr in (precisions, recalls, f1s):
        np.nan_to_num(arr, copy=False, nan=zero_division)

    return tn, tp, fn, fp, accuracy, supports, precisions, recalls, f1s
```

### Describe alternatives you've considered, if relevant

Can someone from original classification_report authors take a look please why it's taking so long?

### Additional context

I have the latest versions of scikit-learn and numba to this date, an Intel CPU with AVX.

COMMENT:
numba no, but passing optional n_classes makes sense.

Labels: Performance, Needs Investigation
⚠️ CI failed on Linux_Runs.pylatest_conda_forge_mkl (last failure: Jul 10, 2024) ⚠️

**CI is still failing on [Linux_Runs.pylatest_conda_forge_mkl](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=68533&view=logs&j=dde5042c-7464-5d47-9507-31bdd2ee0a3a)** (Jul 10, 2024)
- test_fastica_eigh_low_rank_warning[31]

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=69196&view=logs&j=dde5042c-7464-5d47-9507-31bdd2ee0a3a) on Aug 02, 2024

Labels: Bug
⚠️ CI failed on Linux_Runs.pylatest_conda_forge_mkl (last failure: Jul 10, 2024) ⚠️

**CI is still failing on [Linux_Runs.pylatest_conda_forge_mkl](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=68533&view=logs&j=dde5042c-7464-5d47-9507-31bdd2ee0a3a)** (Jul 10, 2024)
- test_fastica_eigh_low_rank_warning[31]

COMMENT:
For future reference, it seems like for some global random seed we get a `ConvergenceWarning` rather than a `UserWarning` about singular values:

```
____________________ test_fastica_eigh_low_rank_warning[31] ____________________
[gw1] linux -- Python 3.11.4 /usr/share/miniconda/envs/testvenv/bin/python

global_random_seed = 31

    def test_fastica_eigh_low_rank_warning(global_random_seed):
        """Test FastICA eigh solver raises warning for low-rank data."""
        rng = np.random.RandomState(global_random_seed)
        A = rng.randn(10, 2)
        X = A @ A.T
        ica = FastICA(random_state=0, whiten="unit-variance", whiten_solver="eigh")
        msg = "There are some small singular values"
>       with pytest.warns(UserWarning, match=msg):
E       Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) matching the regex were emitted.
E        Regex: There are some small singular values
E        Emitted warnings: [ ConvergenceWarning('FastICA did not converge. Consider increasing tolerance or the maximum number of iterations.')]

A          = array([[-0.41475721, -0.33336867],
       [ 0.08109199, -0.79102695],
       [-0.21859967, -0.76319684],
       [-0.77...-0.91879129],
       [ 1.19943449, -0.34137424],
       [-1.75860731,  0.05111747],
       [-0.57192899, -0.70056604]])
X          = array([[ 2.83158216e-01,  2.30070116e-01,  3.45091706e-01,
        -2.94201294e-01,  3.21336537e-01, -7.56222642e-02,
... 4.63831401e-01, -7.26324674e-02,
         1.20573385e+00, -4.46836158e-01,  9.69987343e-01,
         8.17895550e-01]])
global_random_seed = 31
ica        = FastICA(random_state=0, whiten_solver='eigh')
msg        = 'There are some small singular values'
rng        = RandomState(MT19937) at 0x7F71B6CA6940

../1/s/sklearn/decomposition/tests/test_fastica.py:450: Failed
```

Labels: Bug
⚠️ CI failed on Linux_Runs.pylatest_conda_forge_mkl (last failure: Jul 10, 2024) ⚠️

**CI is still failing on [Linux_Runs.pylatest_conda_forge_mkl](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=68533&view=logs&j=dde5042c-7464-5d47-9507-31bdd2ee0a3a)** (Jul 10, 2024)
- test_fastica_eigh_low_rank_warning[31]

COMMENT:
The last error is about `test_minibatch_sensible_reassign[34]`: Edit 3 weeks after creating the comment, the `test_minibatch_sensible_reassign` should now have been fixed by https://github.com/scikit-learn/scikit-learn/pull/29278 :crossed_fingers: 

```
_____________________ test_minibatch_sensible_reassign[34] _____________________
[gw0] linux -- Python 3.11.9 /usr/share/miniconda/envs/testvenv/bin/python

global_random_seed = 34
    def test_minibatch_sensible_reassign(global_random_seed):
        # check that identical initial clusters are reassigned
        # also a regression test for when there are more desired reassignments than
        # samples.
        zeroed_X, true_labels = make_blobs(
            n_samples=100, centers=5, random_state=global_random_seed
        )
        zeroed_X[::2, :] = 0
    
        km = MiniBatchKMeans(
            n_clusters=20, batch_size=10, random_state=global_random_seed, init="random"
        ).fit(zeroed_X)
        # there should not be too many exact zero cluster centers
>       assert km.cluster_centers_.any(axis=1).sum() > 10
E       AssertionError

global_random_seed = 34
km         = MiniBatchKMeans(batch_size=10, init='random', n_clusters=20, random_state=34)
true_labels = array([3, 0, 2, 4, 1, 0, 2, 3, 0, 1, 4, 2, 2, 0, 2, 4, 3, 4, 2, 3, 3, 4,
       2, 2, 1, 4, 3, 4, 3, 1, 1, 1, 4, 1, 4,...3,
       4, 1, 0, 3, 3, 4, 3, 2, 2, 2, 0, 2, 2, 4, 4, 0, 4, 2, 3, 0, 2, 1,
       3, 0, 1, 2, 0, 3, 1, 0, 0, 4, 2, 4])
zeroed_X   = array([[  0.        ,   0.        ],
       [ -9.93451852,   5.17769196],
       [  0.        ,   0.        ],
       ...     ],
       [ -5.62221677,   0.01380172],
       [  0.        ,   0.        ],
       [ -6.95512025,  -1.26162786]])

../1/s/sklearn/cluster/tests/test_k_means.py:440: AssertionError
```

Labels: Bug
⚠️ CI failed on Linux_Runs.pylatest_conda_forge_mkl (last failure: Jul 10, 2024) ⚠️

**CI is still failing on [Linux_Runs.pylatest_conda_forge_mkl](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=68533&view=logs&j=dde5042c-7464-5d47-9507-31bdd2ee0a3a)** (Jul 10, 2024)
- test_fastica_eigh_low_rank_warning[31]

COMMENT:
For reference, the `test_fastica_eigh_low_rank_warning[31]` failure happened again last night with the same symptoms as described above (`ConvergenceWarning` instead of `UserWarning`).

Labels: Bug
HDBSCAN Ongoing Work

## Introduction

This is a (hopefully) exhaustive list of ongoing/future work for HDBSCAN. These have all been discussed and are considered wanted, but some still require thorough investigation (especially heuristic evaluations).

## Priority List

The higher priority items appear earlier in this list.

- [ ] https://github.com/scikit-learn/scikit-learn/pull/26888
    - [x] https://github.com/scikit-learn/scikit-learn/pull/25914
    - [ ] Finalize `{KD, Ball}Tree` API to avoid writing custom dispatcher
- [ ] Support `np.nan` in Cython implementation for sparse matrices
    - [x] https://github.com/scikit-learn/scikit-learn/pull/26889
- [ ] [Reintroduce `Boruvka` algorithm](https://github.com/scikit-learn/scikit-learn/pull/27572) (removed in https://github.com/scikit-learn/scikit-learn/commit/b7736ef6db1650ba4c3d8d830348aacfe5589015)
- [ ] Implement PWD backend for weighted `argkmin` in medoid calculation
- [ ] Investigate PWD backend for `mst_from_*` functions in `_linkage.pyx`
- [ ] Investigate PWD backend for `_reachability.pyx`
- [ ] Benchmark KD vs Ball Tree efficiency
- [ ] Add consistent threading semantics to enable `prange`, e.g. in `_reachability.pyx`
- [x] Improve partition strategy in `_reachability.pyx` (cf. https://github.com/scikit-learn/scikit-learn/pull/24701#discussion_r1035775220)
- [ ] Add support for `np.inf` values when `metric=='precomputed'` and `X` is sparse.

COMMENT:
Has someone aligned with the devs from https://github.com/scikit-learn-contrib/hdbscan?
What is the long term path forward here? Will the contrib package be archived in the future? Will they help making the scikit-learn implementation roughly equivalent in features?

@lmcinnes friendly ping. Your opinion is highly appreciated.

Labels: New Feature, Moderate, module:cluster, cython
HDBSCAN Ongoing Work

## Introduction

This is a (hopefully) exhaustive list of ongoing/future work for HDBSCAN. These have all been discussed and are considered wanted, but some still require thorough investigation (especially heuristic evaluations).

## Priority List

The higher priority items appear earlier in this list.

- [ ] https://github.com/scikit-learn/scikit-learn/pull/26888
    - [x] https://github.com/scikit-learn/scikit-learn/pull/25914
    - [ ] Finalize `{KD, Ball}Tree` API to avoid writing custom dispatcher
- [ ] Support `np.nan` in Cython implementation for sparse matrices
    - [x] https://github.com/scikit-learn/scikit-learn/pull/26889
- [ ] [Reintroduce `Boruvka` algorithm](https://github.com/scikit-learn/scikit-learn/pull/27572) (removed in https://github.com/scikit-learn/scikit-learn/commit/b7736ef6db1650ba4c3d8d830348aacfe5589015)
- [ ] Implement PWD backend for weighted `argkmin` in medoid calculation
- [ ] Investigate PWD backend for `mst_from_*` functions in `_linkage.pyx`
- [ ] Investigate PWD backend for `_reachability.pyx`
- [ ] Benchmark KD vs Ball Tree efficiency
- [ ] Add consistent threading semantics to enable `prange`, e.g. in `_reachability.pyx`
- [x] Improve partition strategy in `_reachability.pyx` (cf. https://github.com/scikit-learn/scikit-learn/pull/24701#discussion_r1035775220)
- [ ] Add support for `np.inf` values when `metric=='precomputed'` and `X` is sparse.

COMMENT:
The scikit-learn-contrib package is essentially in a pure maintenance mode right now; just trying to make sure it doesn't break going forward. I would hope that eventually it can just be archived. I think the main features that really needs to be in place before thinking about retiring the sklearn-contrib version is the Dual-tree Boruvka algorithm. I have offered to assist in that if needed (but @Micky774 seems to be more than capable).

Labels: New Feature, Moderate, module:cluster, cython