Talip7/scikit-learn-issues-multilabel huggingface dataset:
Create a classification report using a confusion matrix as input

### Describe the workflow you want to enable

I had some results stored in the form of a confusion matrix for some experiments. However, I wanted to use `classification_report` to easily compute all the metrics.

### Describe your proposed solution

My solution consists of creating new `y_true` and `y_pred` vectors and using those to then call `classification_report`.

```python
def classification_report_from_confusion_matrix(confusion_matrix, **args):
    y_true = np.zeros(np.sum(confusion_matrix), dtype=int)
    y_pred = np.copy(y_true)

    i = 0
    for target in range(len(confusion_matrix)):
        for pred in range(len(confusion_matrix)):
            n = confusion_matrix[target][pred]
            y_true[i:i+n] = target
            y_pred[i:i+n] = pred
            i += n

    return metrics.classification_report(y_true, y_pred, **args)
```

### Describe alternatives you've considered, if relevant

I first tried using lists and appending values, but the `numpy` version performs faster, although slightly less readable:

```python
def classification_report_from_confusion_matrix(confusion_matrix, **args):

    y_true = []
    y_pred = []

    for gt in range(len(total)):
        for pred in range(len(total)):
            y_true += [gt]*total[gt][pred]
            y_pred += [pred]*total[gt][pred]

    return metrics.classification_report(y_true , y_pred, **args)
```

### Additional context

I was comparing two approaches:
* **Two separate models**, each for different tasks
* One **single model** incorporating both tasks

Two compare them, I built a confusion matrix of the two separate models and concatenated them so that the final confusion matrix would be similar to the single model.
I used this function to generate the classification report of the **two separate models** after concatenating their confusion matrices.
In the case of the **single** model, I could use `classification_report` directly.

COMMENT:
In this case, it is annoying to have two functions that do the same. I assume that having a class with two class methods (e.g. `from_predictions` and `from_confusion_matrix`).

Then, I am not sure that this is worth creating this feature since you can always write a very small helper function to generate the predictions. For instance, it would give something like this:

```python
import itertools


def generate_gt_pred(cm):
    y_true, y_pred = [], []
    for gt_class, row in enumerate(cm):
        y_true.append([gt_class] * row.sum())
        y_pred.append(list(itertools.chain.from_iterable(
            [
                [idx] * val for idx, val in enumerate(row)
            ]
        )))
    return np.array(y_true).ravel(), np.array(y_pred).ravel()
```

Then you can create the report with:

```python
classification_report(*generate_gt_pred(confusion_matrix))
```

Labels: Documentation
Create a classification report using a confusion matrix as input

### Describe the workflow you want to enable

I had some results stored in the form of a confusion matrix for some experiments. However, I wanted to use `classification_report` to easily compute all the metrics.

### Describe your proposed solution

My solution consists of creating new `y_true` and `y_pred` vectors and using those to then call `classification_report`.

```python
def classification_report_from_confusion_matrix(confusion_matrix, **args):
    y_true = np.zeros(np.sum(confusion_matrix), dtype=int)
    y_pred = np.copy(y_true)

    i = 0
    for target in range(len(confusion_matrix)):
        for pred in range(len(confusion_matrix)):
            n = confusion_matrix[target][pred]
            y_true[i:i+n] = target
            y_pred[i:i+n] = pred
            i += n

    return metrics.classification_report(y_true, y_pred, **args)
```

### Describe alternatives you've considered, if relevant

I first tried using lists and appending values, but the `numpy` version performs faster, although slightly less readable:

```python
def classification_report_from_confusion_matrix(confusion_matrix, **args):

    y_true = []
    y_pred = []

    for gt in range(len(total)):
        for pred in range(len(total)):
            y_true += [gt]*total[gt][pred]
            y_pred += [pred]*total[gt][pred]

    return metrics.classification_report(y_true , y_pred, **args)
```

### Additional context

I was comparing two approaches:
* **Two separate models**, each for different tasks
* One **single model** incorporating both tasks

Two compare them, I built a confusion matrix of the two separate models and concatenated them so that the final confusion matrix would be similar to the single model.
I used this function to generate the classification report of the **two separate models** after concatenating their confusion matrices.
In the case of the **single** model, I could use `classification_report` directly.

COMMENT:
Oh, I see that you already propose the solution sorry.
I am still not sure this is worth it since it is quite easy to write your own helper or wrapper.

ping @thomasjpfan to discuss this in the triage meeting later today.

Labels: Documentation
Create a classification report using a confusion matrix as input

### Describe the workflow you want to enable

I had some results stored in the form of a confusion matrix for some experiments. However, I wanted to use `classification_report` to easily compute all the metrics.

### Describe your proposed solution

My solution consists of creating new `y_true` and `y_pred` vectors and using those to then call `classification_report`.

```python
def classification_report_from_confusion_matrix(confusion_matrix, **args):
    y_true = np.zeros(np.sum(confusion_matrix), dtype=int)
    y_pred = np.copy(y_true)

    i = 0
    for target in range(len(confusion_matrix)):
        for pred in range(len(confusion_matrix)):
            n = confusion_matrix[target][pred]
            y_true[i:i+n] = target
            y_pred[i:i+n] = pred
            i += n

    return metrics.classification_report(y_true, y_pred, **args)
```

### Describe alternatives you've considered, if relevant

I first tried using lists and appending values, but the `numpy` version performs faster, although slightly less readable:

```python
def classification_report_from_confusion_matrix(confusion_matrix, **args):

    y_true = []
    y_pred = []

    for gt in range(len(total)):
        for pred in range(len(total)):
            y_true += [gt]*total[gt][pred]
            y_pred += [pred]*total[gt][pred]

    return metrics.classification_report(y_true , y_pred, **args)
```

### Additional context

I was comparing two approaches:
* **Two separate models**, each for different tasks
* One **single model** incorporating both tasks

Two compare them, I built a confusion matrix of the two separate models and concatenated them so that the final confusion matrix would be similar to the single model.
I used this function to generate the classification report of the **two separate models** after concatenating their confusion matrices.
In the case of the **single** model, I could use `classification_report` directly.

COMMENT:
> In this case, it is annoying to have two functions that do the same. I assume that having a class with two class methods (e.g. `from_predictions` and `from_confusion_matrix`).

I had thought about this and honestly think this is the main issue, adapting it not to have two functions to do the same would probably not be backward-compatible.

Labels: Documentation
Create a classification report using a confusion matrix as input

### Describe the workflow you want to enable

I had some results stored in the form of a confusion matrix for some experiments. However, I wanted to use `classification_report` to easily compute all the metrics.

### Describe your proposed solution

My solution consists of creating new `y_true` and `y_pred` vectors and using those to then call `classification_report`.

```python
def classification_report_from_confusion_matrix(confusion_matrix, **args):
    y_true = np.zeros(np.sum(confusion_matrix), dtype=int)
    y_pred = np.copy(y_true)

    i = 0
    for target in range(len(confusion_matrix)):
        for pred in range(len(confusion_matrix)):
            n = confusion_matrix[target][pred]
            y_true[i:i+n] = target
            y_pred[i:i+n] = pred
            i += n

    return metrics.classification_report(y_true, y_pred, **args)
```

### Describe alternatives you've considered, if relevant

I first tried using lists and appending values, but the `numpy` version performs faster, although slightly less readable:

```python
def classification_report_from_confusion_matrix(confusion_matrix, **args):

    y_true = []
    y_pred = []

    for gt in range(len(total)):
        for pred in range(len(total)):
            y_true += [gt]*total[gt][pred]
            y_pred += [pred]*total[gt][pred]

    return metrics.classification_report(y_true , y_pred, **args)
```

### Additional context

I was comparing two approaches:
* **Two separate models**, each for different tasks
* One **single model** incorporating both tasks

Two compare them, I built a confusion matrix of the two separate models and concatenated them so that the final confusion matrix would be similar to the single model.
I used this function to generate the classification report of the **two separate models** after concatenating their confusion matrices.
In the case of the **single** model, I could use `classification_report` directly.

COMMENT:
> adapting it not to have two functions to do the same would probably not be backward-compatible.

With a single function, we would have to add a parameter `cm` for instance. Then, either you set `cm` or the `y_true`/`y_pred`. This would be backward-compatible but I don't really like that you can do that many things with a single function.

Labels: Documentation
Create a classification report using a confusion matrix as input

### Describe the workflow you want to enable

I had some results stored in the form of a confusion matrix for some experiments. However, I wanted to use `classification_report` to easily compute all the metrics.

### Describe your proposed solution

My solution consists of creating new `y_true` and `y_pred` vectors and using those to then call `classification_report`.

```python
def classification_report_from_confusion_matrix(confusion_matrix, **args):
    y_true = np.zeros(np.sum(confusion_matrix), dtype=int)
    y_pred = np.copy(y_true)

    i = 0
    for target in range(len(confusion_matrix)):
        for pred in range(len(confusion_matrix)):
            n = confusion_matrix[target][pred]
            y_true[i:i+n] = target
            y_pred[i:i+n] = pred
            i += n

    return metrics.classification_report(y_true, y_pred, **args)
```

### Describe alternatives you've considered, if relevant

I first tried using lists and appending values, but the `numpy` version performs faster, although slightly less readable:

```python
def classification_report_from_confusion_matrix(confusion_matrix, **args):

    y_true = []
    y_pred = []

    for gt in range(len(total)):
        for pred in range(len(total)):
            y_true += [gt]*total[gt][pred]
            y_pred += [pred]*total[gt][pred]

    return metrics.classification_report(y_true , y_pred, **args)
```

### Additional context

I was comparing two approaches:
* **Two separate models**, each for different tasks
* One **single model** incorporating both tasks

Two compare them, I built a confusion matrix of the two separate models and concatenated them so that the final confusion matrix would be similar to the single model.
I used this function to generate the classification report of the **two separate models** after concatenating their confusion matrices.
In the case of the **single** model, I could use `classification_report` directly.

COMMENT:
In the triage meeting, we decided that we will not implement this feature as a new function or class.

However, we could find one example where we could implement the small helper function that you provided. I can see a couple of candidates in the bottom gallery examples of the `classification_report` API documentation: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html#examples-using-sklearn-metrics-classification-report

@joaocmd would you like to propose a PR to modify one of these examples?

Labels: Documentation
Create a classification report using a confusion matrix as input

### Describe the workflow you want to enable

I had some results stored in the form of a confusion matrix for some experiments. However, I wanted to use `classification_report` to easily compute all the metrics.

### Describe your proposed solution

My solution consists of creating new `y_true` and `y_pred` vectors and using those to then call `classification_report`.

```python
def classification_report_from_confusion_matrix(confusion_matrix, **args):
    y_true = np.zeros(np.sum(confusion_matrix), dtype=int)
    y_pred = np.copy(y_true)

    i = 0
    for target in range(len(confusion_matrix)):
        for pred in range(len(confusion_matrix)):
            n = confusion_matrix[target][pred]
            y_true[i:i+n] = target
            y_pred[i:i+n] = pred
            i += n

    return metrics.classification_report(y_true, y_pred, **args)
```

### Describe alternatives you've considered, if relevant

I first tried using lists and appending values, but the `numpy` version performs faster, although slightly less readable:

```python
def classification_report_from_confusion_matrix(confusion_matrix, **args):

    y_true = []
    y_pred = []

    for gt in range(len(total)):
        for pred in range(len(total)):
            y_true += [gt]*total[gt][pred]
            y_pred += [pred]*total[gt][pred]

    return metrics.classification_report(y_true , y_pred, **args)
```

### Additional context

I was comparing two approaches:
* **Two separate models**, each for different tasks
* One **single model** incorporating both tasks

Two compare them, I built a confusion matrix of the two separate models and concatenated them so that the final confusion matrix would be similar to the single model.
I used this function to generate the classification report of the **two separate models** after concatenating their confusion matrices.
In the case of the **single** model, I could use `classification_report` directly.

COMMENT:
@glemaitre I would like to do so, will get on to work 😄.

Labels: Documentation
TfidfVectorizer binary parameter

### Describe the issue linked to the documentation

The `TfidfVectorizer` parameter `binary` is described as:

> **binary bool, default=False**
> If True, all non-zero term counts are set to 1. This does not mean outputs will have only 0/1 values, only that the tf term in tf-idf is binary. (Set idf and normalization to False to get 0/1 outputs).

Yet there's no parameter called `normalization` so it's not clear what parameters the last comment is referring to in order to get 0/1 outputs.

### Suggest a potential alternative/fix

_No response_

COMMENT:
There is a `norm` parameter but if I understand correctly it is not what they mean.
What makes more sense is that they refer to `smooth_idf` which is true by default.

I would like a second opinion on this though before we suggest fixes

Labels: Documentation, module:feature_extraction
TfidfVectorizer binary parameter

### Describe the issue linked to the documentation

The `TfidfVectorizer` parameter `binary` is described as:

> **binary bool, default=False**
> If True, all non-zero term counts are set to 1. This does not mean outputs will have only 0/1 values, only that the tf term in tf-idf is binary. (Set idf and normalization to False to get 0/1 outputs).

Yet there's no parameter called `normalization` so it's not clear what parameters the last comment is referring to in order to get 0/1 outputs.

### Suggest a potential alternative/fix

_No response_

COMMENT:
I am thinking this issue is closed, since #26697

Labels: Documentation, module:feature_extraction
TfidfVectorizer binary parameter

### Describe the issue linked to the documentation

The `TfidfVectorizer` parameter `binary` is described as:

> **binary bool, default=False**
> If True, all non-zero term counts are set to 1. This does not mean outputs will have only 0/1 values, only that the tf term in tf-idf is binary. (Set idf and normalization to False to get 0/1 outputs).

Yet there's no parameter called `normalization` so it's not clear what parameters the last comment is referring to in order to get 0/1 outputs.

### Suggest a potential alternative/fix

_No response_

COMMENT:
Thanks for spotting this!

Labels: Documentation, module:feature_extraction
Introduction of `Waiting for Second Reviewer` tag

## Motivation
PRs commonly fall into a state where they have one approval from an initial reviewer, however no attention from any secondary reviewers. Arguably the most effective mechanism of dissolving PR backlog is exactly to target these sorts of PRs, since the initial approval implies that the underlying quality/status of the work is not too far from mergeable, and the bottleneck lies with a lack of reviewer attention rather than the contributors/authors. Providing a specific tag for this will hopefully make it easier for core developers to find these PRs and push for final merger.

## Current Solution
Currently we have the `Waiting for Reviewer` tag, applied to ~40 PRs, which is generally used to indicate/request extra attention to a PR. This, in combination with a filter for PRs with existing approval ([for example](https://github.com/scikit-learn/scikit-learn/pulls?q=is%3Aopen+is%3Apr+label%3A%22Waiting+for+Reviewer%22+review%3Aapproved)) then results in PRs that, on paper, should also be mergeable with a single extra approval. However there are some flaws to this:

1. The `Waiting for Reviewer` tag is a bit redundant and ill-defined. Every PR, short of approval, is in some sense "waiting for a reviewer", unless it is very specifically waiting on an author to implement changes from existing review. This results in the `Waiting for Reviewer` tag's meaning being muddled.
2. Labeling with the `Waiting for Reviewer` tag is manual. While it could be automated, as it exists right now there is little sense in doing so since -- again -- every PR is in some sense "waiting for a reviewer". This means that coverage is not guaranteed to be thorough and that the label is not exhaustive/surjective.
3. Some PRs which satisfy the above filter cannot be easily merged by another reviewer. In particular, if a PR with the `Waiting for Reviewer` tag and one approval has a review which has requested _changes_ then it would be a false-positive in the sense that the review ought to be continued by the initial reviewer that requested changes once those requests have been addressed.

## Proposed Solution
A very specific `Waiting for Second Reviewer` tag. A PR is eligible for this tag when:
1. It has been approved by one reviewer with commit rights.
2. It has NOT been reviewed by a second reviewer with commit rights (including reviews with `approval`, `comments`, and `changes requested`)

The narrow scope of the tag would also make it relatively simple to automate (hopefully 😄).

## Next Steps (?)
I wonder if we should formalize/clarify the role of `Waiting for Reviewer`, with or without the acceptance of this new tag.

Note: This was discussed in the Drafting Meeting on 10/07/2022: [notes here](https://github.com/Micky774/administrative/blob/drafting_notes/drafting_meetings/2022-10-07)

COMMENT:
I think this makes sense. I have added the tag and we can start using it and see how it goes.

Labels: workflow
Introduction of `Waiting for Second Reviewer` tag

## Motivation
PRs commonly fall into a state where they have one approval from an initial reviewer, however no attention from any secondary reviewers. Arguably the most effective mechanism of dissolving PR backlog is exactly to target these sorts of PRs, since the initial approval implies that the underlying quality/status of the work is not too far from mergeable, and the bottleneck lies with a lack of reviewer attention rather than the contributors/authors. Providing a specific tag for this will hopefully make it easier for core developers to find these PRs and push for final merger.

## Current Solution
Currently we have the `Waiting for Reviewer` tag, applied to ~40 PRs, which is generally used to indicate/request extra attention to a PR. This, in combination with a filter for PRs with existing approval ([for example](https://github.com/scikit-learn/scikit-learn/pulls?q=is%3Aopen+is%3Apr+label%3A%22Waiting+for+Reviewer%22+review%3Aapproved)) then results in PRs that, on paper, should also be mergeable with a single extra approval. However there are some flaws to this:

1. The `Waiting for Reviewer` tag is a bit redundant and ill-defined. Every PR, short of approval, is in some sense "waiting for a reviewer", unless it is very specifically waiting on an author to implement changes from existing review. This results in the `Waiting for Reviewer` tag's meaning being muddled.
2. Labeling with the `Waiting for Reviewer` tag is manual. While it could be automated, as it exists right now there is little sense in doing so since -- again -- every PR is in some sense "waiting for a reviewer". This means that coverage is not guaranteed to be thorough and that the label is not exhaustive/surjective.
3. Some PRs which satisfy the above filter cannot be easily merged by another reviewer. In particular, if a PR with the `Waiting for Reviewer` tag and one approval has a review which has requested _changes_ then it would be a false-positive in the sense that the review ought to be continued by the initial reviewer that requested changes once those requests have been addressed.

## Proposed Solution
A very specific `Waiting for Second Reviewer` tag. A PR is eligible for this tag when:
1. It has been approved by one reviewer with commit rights.
2. It has NOT been reviewed by a second reviewer with commit rights (including reviews with `approval`, `comments`, and `changes requested`)

The narrow scope of the tag would also make it relatively simple to automate (hopefully 😄).

## Next Steps (?)
I wonder if we should formalize/clarify the role of `Waiting for Reviewer`, with or without the acceptance of this new tag.

Note: This was discussed in the Drafting Meeting on 10/07/2022: [notes here](https://github.com/Micky774/administrative/blob/drafting_notes/drafting_meetings/2022-10-07)

COMMENT:
What about a "Waiting for merge" for those PRs like eg https://github.com/scikit-learn/scikit-learn/pull/20331... :innocent:

Labels: workflow