Talip7/scikit-learn-issues-multilabel huggingface dataset:
GridSearchCV.verbose documentation does not match actual behavior (values >3, >10)

### Describe the issue linked to the documentation

[`GridSearchCV` docs ](https://scikit-learn.org/1.7/modules/generated/sklearn.model_selection.GridSearchCV.html) state

> **verbose** int
> Controls the verbosity: the higher, the more messages.
> \>1 : the computation time for each fold and parameter candidate is displayed;
> \>2 : the score is also displayed;
> \>3 : the fold and candidate parameter indexes are also displayed together with the starting time of the computation.

In reality, this param is passed to joblib [here](https://github.com/scikit-learn/scikit-learn/blob/d3898d9d57aeb1e960d266613a2e31b07bca39d7/sklearn/model_selection/_search.py#L1009), whose [docs](https://joblib.readthedocs.io/en/latest/generated/joblib.Parallel.html) state:
> **verbose**: int, default=0
> The verbosity level: if non zero, progress messages are printed. Above 50, the output is sent to stdout. The frequency of the messages increases with the verbosity level. If it more than 10, all iterations are reported.

Testing with the MRE below shows that 0 is silent, 1 prints one line, 2 prints a line for each CV split. 3-9 are all the same. 10-inf are all the same

### MRE:
```python
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

X, y = make_classification()

clf = LogisticRegression(max_iter=100)

param_grid = {"C": [1, 2]}

for v in [0, 1, 2, 3, 9, 10, 1000]:
    print(f"\n=======  verbose={v}  ======= ")
    gs = GridSearchCV(
        clf,
        param_grid,
        cv=2,
        n_jobs=1,
        verbose=v,
    )
    gs.fit(X, y)
```

It is possible taht the same issue exists elsewhere, I've not looked thoroughly

### Suggest a potential alternative/fix

Update docs to match observed behaviour, e.g.:

> **verbose** int
> Controls the verbosity: the higher, the more messages.
> 0 : no messages;
> 1 : a single line describing the number of folds, the number of candidates, and the number of fits;
> 2 : the above, plus when each fold ends, the computation time for that fold and parameter candidate is displayed;
> 3-9 : the above, plus the fold index and the score is also displayed;
> ≥10 : an additional message for the start of each fold, and the candidate parameter indexes are also displayed.

COMMENT:
Hi @StefanieSenger , for the clarification.
Yes, I’m interested in making a documentation-only PR.
I’ll update the docs for both GridSearchCV and RandomizedSearchCV, keep it lean, and document only the observed behavior (including verbose=0 and the verbose > 10 case), without backend-specific guarantees.

Labels: Documentation
GridSearchCV.verbose documentation does not match actual behavior (values >3, >10)

### Describe the issue linked to the documentation

[`GridSearchCV` docs ](https://scikit-learn.org/1.7/modules/generated/sklearn.model_selection.GridSearchCV.html) state

> **verbose** int
> Controls the verbosity: the higher, the more messages.
> \>1 : the computation time for each fold and parameter candidate is displayed;
> \>2 : the score is also displayed;
> \>3 : the fold and candidate parameter indexes are also displayed together with the starting time of the computation.

In reality, this param is passed to joblib [here](https://github.com/scikit-learn/scikit-learn/blob/d3898d9d57aeb1e960d266613a2e31b07bca39d7/sklearn/model_selection/_search.py#L1009), whose [docs](https://joblib.readthedocs.io/en/latest/generated/joblib.Parallel.html) state:
> **verbose**: int, default=0
> The verbosity level: if non zero, progress messages are printed. Above 50, the output is sent to stdout. The frequency of the messages increases with the verbosity level. If it more than 10, all iterations are reported.

Testing with the MRE below shows that 0 is silent, 1 prints one line, 2 prints a line for each CV split. 3-9 are all the same. 10-inf are all the same

### MRE:
```python
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

X, y = make_classification()

clf = LogisticRegression(max_iter=100)

param_grid = {"C": [1, 2]}

for v in [0, 1, 2, 3, 9, 10, 1000]:
    print(f"\n=======  verbose={v}  ======= ")
    gs = GridSearchCV(
        clf,
        param_grid,
        cv=2,
        n_jobs=1,
        verbose=v,
    )
    gs.fit(X, y)
```

It is possible taht the same issue exists elsewhere, I've not looked thoroughly

### Suggest a potential alternative/fix

Update docs to match observed behaviour, e.g.:

> **verbose** int
> Controls the verbosity: the higher, the more messages.
> 0 : no messages;
> 1 : a single line describing the number of folds, the number of candidates, and the number of fits;
> 2 : the above, plus when each fold ends, the computation time for that fold and parameter candidate is displayed;
> 3-9 : the above, plus the fold index and the score is also displayed;
> ≥10 : an additional message for the start of each fold, and the candidate parameter indexes are also displayed.

COMMENT:
> I'd understood it looped over each param candidate and then over the splits

That is indeed true. I had thought the wrong way around but after checking the code I see you're right. The parallel execution of the `_fit_and_score` (li 999-1015) iterates over the params as the outer loop and then fits and scores on one split at the time. So the output you suggested would indeed be possible.

> OK let's leave it as a docs change for now as you suggest 😃

I'm happy with this, too.

Labels: Documentation
DOC Supervised clustering metrics specify array input dtype

Several of our supervised clustering metrics specify/suggest (??) that the array input should be of type `int` or `integral`, e.g.:

`pair_confusion_matrix`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L212-L216

`rand_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L298-L302

`mutual_info_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L858-L864

`fowlkes_mallows_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L1221-L1225

Looking at the code/testing a few - we don't seem to enforce this. I also have not seen specification of dtype of array input much elsewhere in the codebase. Is this dytpe specification needed in the docs?

Edit: I mean is it worth clarifying it is a suggestion and not enforced?

COMMENT:
Hi! I’m interested in working on this. Before I begin: 
1) Should we make the docs reflect that non-integral label types are accepted? 
2) Or should the code enforce integer label arrays?

Labels: RFC
DOC Supervised clustering metrics specify array input dtype

Several of our supervised clustering metrics specify/suggest (??) that the array input should be of type `int` or `integral`, e.g.:

`pair_confusion_matrix`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L212-L216

`rand_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L298-L302

`mutual_info_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L858-L864

`fowlkes_mallows_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L1221-L1225

Looking at the code/testing a few - we don't seem to enforce this. I also have not seen specification of dtype of array input much elsewhere in the codebase. Is this dytpe specification needed in the docs?

Edit: I mean is it worth clarifying it is a suggestion and not enforced?

COMMENT:
Hello this is my first contribution, I would be very enthused to work on this .

Labels: RFC
DOC Supervised clustering metrics specify array input dtype

Several of our supervised clustering metrics specify/suggest (??) that the array input should be of type `int` or `integral`, e.g.:

`pair_confusion_matrix`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L212-L216

`rand_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L298-L302

`mutual_info_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L858-L864

`fowlkes_mallows_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L1221-L1225

Looking at the code/testing a few - we don't seem to enforce this. I also have not seen specification of dtype of array input much elsewhere in the codebase. Is this dytpe specification needed in the docs?

Edit: I mean is it worth clarifying it is a suggestion and not enforced?

COMMENT:
Can I work on this ?

Labels: RFC
DOC Supervised clustering metrics specify array input dtype

Several of our supervised clustering metrics specify/suggest (??) that the array input should be of type `int` or `integral`, e.g.:

`pair_confusion_matrix`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L212-L216

`rand_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L298-L302

`mutual_info_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L858-L864

`fowlkes_mallows_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L1221-L1225

Looking at the code/testing a few - we don't seem to enforce this. I also have not seen specification of dtype of array input much elsewhere in the codebase. Is this dytpe specification needed in the docs?

Edit: I mean is it worth clarifying it is a suggestion and not enforced?

COMMENT:
I’d like to work on this issue.

Labels: RFC
DOC Supervised clustering metrics specify array input dtype

Several of our supervised clustering metrics specify/suggest (??) that the array input should be of type `int` or `integral`, e.g.:

`pair_confusion_matrix`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L212-L216

`rand_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L298-L302

`mutual_info_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L858-L864

`fowlkes_mallows_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L1221-L1225

Looking at the code/testing a few - we don't seem to enforce this. I also have not seen specification of dtype of array input much elsewhere in the codebase. Is this dytpe specification needed in the docs?

Edit: I mean is it worth clarifying it is a suggestion and not enforced?

COMMENT:
Some of the clustering metrics raise a warning if the input type is continuous:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L63-L69

It was added in #17911. (I am wondering why we would only warn and not error here though.)

Since we expect integers, I think it is consistent to mention this in the docs, but then the code needs to comply to this.

> Looking at the code/testing a few - we don't seem to enforce this. 

Is it, that there were only warnings, or did those you had tested not even warn? I would expect an error here and I wonder if we should error instead of warn.

It would be an interesting next step to check all the clustering metrics from this module and see if all that should raise a warning actually use `check_clusterings()` internally. It is probably an easy issue to work on, but not an issue for a first time contributor. Do we know someone with a bit of experience in the repo who'd be happy to take care of this?

Labels: RFC
DOC Supervised clustering metrics specify array input dtype

Several of our supervised clustering metrics specify/suggest (??) that the array input should be of type `int` or `integral`, e.g.:

`pair_confusion_matrix`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L212-L216

`rand_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L298-L302

`mutual_info_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L858-L864

`fowlkes_mallows_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L1221-L1225

Looking at the code/testing a few - we don't seem to enforce this. I also have not seen specification of dtype of array input much elsewhere in the codebase. Is this dytpe specification needed in the docs?

Edit: I mean is it worth clarifying it is a suggestion and not enforced?

COMMENT:
> Some of the clustering metrics raise a warning if the input type is continuous:

Good point I didn't look into that part. However, discrete vs continuous does not seem to depend on dtype. e.g.:

```python
In [7]: from sklearn.utils.multiclass import type_of_target

In [8]: type_of_target(np.array([1.0, 1.0, 0.0, 0.0]))
Out[8]: 'binary'
```

(I checked that dtype of `np.array([1.0, 1.0, 0.0, 0.0])` is float64)

And I just checked with the docstring example, changing the input to float instead of int, and no warning or error for e.g., `rand_score`:

```python
In [1]: from sklearn.metrics.cluster import rand_score
In [4]: import numpy as np
In [5]: rand_score([0, 0, 1, 1.0], np.array([1.0, 1.0, 0, 0]))
Out[5]: 1.0
```

Labels: RFC
DOC Supervised clustering metrics specify array input dtype

Several of our supervised clustering metrics specify/suggest (??) that the array input should be of type `int` or `integral`, e.g.:

`pair_confusion_matrix`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L212-L216

`rand_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L298-L302

`mutual_info_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L858-L864

`fowlkes_mallows_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L1221-L1225

Looking at the code/testing a few - we don't seem to enforce this. I also have not seen specification of dtype of array input much elsewhere in the codebase. Is this dytpe specification needed in the docs?

Edit: I mean is it worth clarifying it is a suggestion and not enforced?

COMMENT:
hi i investigated this issue locally by testing int ,  str and float inputs
i also tested "discrete floats" (like 0.0 and 1.0) to see if the warning is based on type or value

here are the results for all 4 metrics

1. int and str inputs -> works warnings=0
2. discrete floats (0.0, 1.0) -> works warnings=0
3. continuous floats (0.1, 0.2) -> works but triggers the expected "discrete values" warning

this proves that the functions actually check if the **values** are discrete not strictly if the **dtype** is int
so the documentation requirement for `dtype=integral` is definitely too strict

i would be happy to update the docstrings to clarify that **discrete labels** (including strings) are valid inputs

```python
import warnings
from sklearn.metrics import pair_confusion_matrix, rand_score, mutual_info_score, fowlkes_mallows_score

metrics = [
    ("pair_confusion_matrix", pair_confusion_matrix),
    ("rand_score", rand_score),
    ("mutual_info_score", mutual_info_score),
    ("fowlkes_mallows_score", fowlkes_mallows_score),
]

labels = {
    "int": ([1,2,1,2],[1,2,1,2]),
    "str": (['a','b','a','b'],['a','b','a','b']),
    "float_cont": ([0.1,0.2,0.1,0.2],[0.1,0.2,0.1,0.2]),
    "float_disc": ([0.0,0.0,1.0,1.0],[1.0,1.0,0.0,0.0]),
}

for name, metric in metrics:
    print(f"\n--- {name} ---")
    for dtype, (y1, y2) in labels.items():
        with warnings.catch_warnings(record=True) as w:
            warnings.simplefilter("always")
            res = metric(y1, y2)
            print(f"{dtype} -> result={res}, warnings={len(w)}")

Labels: RFC
DOC Supervised clustering metrics specify array input dtype

Several of our supervised clustering metrics specify/suggest (??) that the array input should be of type `int` or `integral`, e.g.:

`pair_confusion_matrix`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L212-L216

`rand_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L298-L302

`mutual_info_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L858-L864

`fowlkes_mallows_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L1221-L1225

Looking at the code/testing a few - we don't seem to enforce this. I also have not seen specification of dtype of array input much elsewhere in the codebase. Is this dytpe specification needed in the docs?

Edit: I mean is it worth clarifying it is a suggestion and not enforced?

COMMENT:
> However, discrete vs continuous does not seem to depend on dtype

Yes, that's true. Providing discrete floats works and doesn't trigger the warning. And the params that are documented to hold the int dtype are actually all starting on "label...", which suggests discreteness.

Now thinking about this, I would go with the suggestion of removing the dtype specification from the docs, without adding any other information constraints on the labels.

This now seems to me as a separate issue, but since we talked about it in here:
Concerning the warning, I was thinking that it was chosen to warn and not to error to keep the code executable, since `type_of_target` is not precise. 

Do you think that  using `type_of_target`  is the right choice to be used in `check_clusterings`? Actually, we only need to check if the inputs are convertible into ints. `check_array` can do that, I think. In this case, we could even error instead of to warn, because check_array can give us a precise answer.

Labels: RFC
DOC Supervised clustering metrics specify array input dtype

Several of our supervised clustering metrics specify/suggest (??) that the array input should be of type `int` or `integral`, e.g.:

`pair_confusion_matrix`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L212-L216

`rand_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L298-L302

`mutual_info_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L858-L864

`fowlkes_mallows_score`:

https://github.com/scikit-learn/scikit-learn/blob/b0ba8b029c298e0cc545206d2df4757be0ec2ac2/sklearn/metrics/cluster/_supervised.py#L1221-L1225

Looking at the code/testing a few - we don't seem to enforce this. I also have not seen specification of dtype of array input much elsewhere in the codebase. Is this dytpe specification needed in the docs?

Edit: I mean is it worth clarifying it is a suggestion and not enforced?

COMMENT:
Thank you @StefanieSenger.

I agree that the warning/error logic in `check_clusterings` is a separate architectural discussion.

Since there is consensus on the documentation part removing the `dtype` specification , I am happy to proceed with only that adjustment to solve this particular issue.

@lucyleeow My previous PR was only for the docs. Do you want me to reopen that one, or create a new PR.

Labels: RFC
Array API spec for `unique_values` does not specify order of unique elements

Noticed when working on #32923

I don't think we need to do anything about this, (and indeed all our tests pass), I just wanted to document this for posterity, in case we encounter any problems later.

Array API spec for `unique_values` does not specify what the order of unique elements should be.
From https://data-apis.org/array-api/latest/API_specification/generated/array_api.unique_values.html#array_api.unique_values :

> The order of unique elements is not specified and may vary between implementations.

Searching for `xp.unique_values` in our codebase, I can see 2 places where order would matter.

In `_check_pos_label_consistency` we do:

https://github.com/scikit-learn/scikit-learn/blob/86acf4547ed8e183cb75c07bcd68ef186a223f06/sklearn/utils/validation.py#L2578-L2582

which assumes a specific order of unique values.

Also in `_unique_np` (used in `_encode.py`), we do:

https://github.com/scikit-learn/scikit-learn/blob/86acf4547ed8e183cb75c07bcd68ef186a223f06/sklearn/utils/_encode.py#L71-L73

assuming NaNs get sorted to the end.

COMMENT:
cc @ogrisel @betatim @lesteve just for visibility.

Labels: Array API
Array API spec for `unique_values` does not specify order of unique elements

Noticed when working on #32923

I don't think we need to do anything about this, (and indeed all our tests pass), I just wanted to document this for posterity, in case we encounter any problems later.

Array API spec for `unique_values` does not specify what the order of unique elements should be.
From https://data-apis.org/array-api/latest/API_specification/generated/array_api.unique_values.html#array_api.unique_values :

> The order of unique elements is not specified and may vary between implementations.

Searching for `xp.unique_values` in our codebase, I can see 2 places where order would matter.

In `_check_pos_label_consistency` we do:

https://github.com/scikit-learn/scikit-learn/blob/86acf4547ed8e183cb75c07bcd68ef186a223f06/sklearn/utils/validation.py#L2578-L2582

which assumes a specific order of unique values.

Also in `_unique_np` (used in `_encode.py`), we do:

https://github.com/scikit-learn/scikit-learn/blob/86acf4547ed8e183cb75c07bcd68ef186a223f06/sklearn/utils/_encode.py#L71-L73

assuming NaNs get sorted to the end.

COMMENT:
> assuming NaNs get sorted to the end.

This is also something that's not specified by the Array API spec for sorting: see the [sorting functions specs](https://data-apis.org/array-api/latest/API_specification/sorting_functions.html)

But sklearn's weighted quantile and scipy's quantile rely on NaNs being sorted at the end, and for now all supported libs do that...

Labels: Array API