Talip7/scikit-learn-issues-multilabel huggingface dataset:
Bug in Calibration Curve Documentation

### Describe the bug

https://scikit-learn.org/stable/auto_examples/calibration/plot_calibration_curve.html

In the calibration curve page, a "scores_df" is generated to showcase supporting model evaluation metrics in addition to the calibration curves.

I noticed that my ROC AUC score was unusually low and noticed that it was consuming Y_PRED instead of Y_PROB. This is incorrect and may confuse users in the future. 

### Steps/Code to Reproduce

Not applicable.

### Expected Results

Expected AUC in the 65-75 range for my application.

### Actual Results

Observed AUC in the 50-55 range instead.

### Versions

```shell
Not really relevant but 1.2.1.
```

COMMENT:
Hi, thanks for reporting this. Would you like to make a Pull Request that fixes this?

Labels: Bug
Bug in Calibration Curve Documentation

### Describe the bug

https://scikit-learn.org/stable/auto_examples/calibration/plot_calibration_curve.html

In the calibration curve page, a "scores_df" is generated to showcase supporting model evaluation metrics in addition to the calibration curves.

I noticed that my ROC AUC score was unusually low and noticed that it was consuming Y_PRED instead of Y_PROB. This is incorrect and may confuse users in the future. 

### Steps/Code to Reproduce

Not applicable.

### Expected Results

Expected AUC in the 65-75 range for my application.

### Actual Results

Observed AUC in the 50-55 range instead.

### Versions

```shell
Not really relevant but 1.2.1.
```

COMMENT:
may I work on this one ?

Labels: Bug
Bug in Calibration Curve Documentation

### Describe the bug

https://scikit-learn.org/stable/auto_examples/calibration/plot_calibration_curve.html

In the calibration curve page, a "scores_df" is generated to showcase supporting model evaluation metrics in addition to the calibration curves.

I noticed that my ROC AUC score was unusually low and noticed that it was consuming Y_PRED instead of Y_PROB. This is incorrect and may confuse users in the future. 

### Steps/Code to Reproduce

Not applicable.

### Expected Results

Expected AUC in the 65-75 range for my application.

### Actual Results

Observed AUC in the 50-55 range instead.

### Versions

```shell
Not really relevant but 1.2.1.
```

COMMENT:
> Hi, thanks for reporting this. Would you like to make a Pull Request that fixes this?

Sure!



> may I work on this one ?

Hey, sorry, if it's okay, I'll take this one.

Labels: Bug
Bug in Calibration Curve Documentation

### Describe the bug

https://scikit-learn.org/stable/auto_examples/calibration/plot_calibration_curve.html

In the calibration curve page, a "scores_df" is generated to showcase supporting model evaluation metrics in addition to the calibration curves.

I noticed that my ROC AUC score was unusually low and noticed that it was consuming Y_PRED instead of Y_PROB. This is incorrect and may confuse users in the future. 

### Steps/Code to Reproduce

Not applicable.

### Expected Results

Expected AUC in the 65-75 range for my application.

### Actual Results

Observed AUC in the 50-55 range instead.

### Versions

```shell
Not really relevant but 1.2.1.
```

COMMENT:
@JonathanJohann  yeah sure, it's yours.

Labels: Bug
Bug in Calibration Curve Documentation

### Describe the bug

https://scikit-learn.org/stable/auto_examples/calibration/plot_calibration_curve.html

In the calibration curve page, a "scores_df" is generated to showcase supporting model evaluation metrics in addition to the calibration curves.

I noticed that my ROC AUC score was unusually low and noticed that it was consuming Y_PRED instead of Y_PROB. This is incorrect and may confuse users in the future. 

### Steps/Code to Reproduce

Not applicable.

### Expected Results

Expected AUC in the 65-75 range for my application.

### Actual Results

Observed AUC in the 50-55 range instead.

### Versions

```shell
Not really relevant but 1.2.1.
```

COMMENT:
Working on this

Labels: Bug
High level documentation of the CI infrastructure

As originally discussed in https://github.com/scikit-learn/scikit-learn/pull/25562#discussion_r1098396646:

I think it might be helpful to give a high level description of our CI somewhere in the doc, both for new contributors and maintainers. In particular, we should summarize the various missions of our CI:

- building and testing PRs to give feedback to contributors and reviewers prior to merging (triggered by commits pushed to the PR branch on the contributor's fork),
- building the stable and dev doc websites (triggered by commits pushed to the `main` and `X.Y.Z` branches on the main repo respectively),
- building and testing scikit-learn against the development versions of its dependencies (triggered as scheduled jobs),
- building, testing, and publishing nightly wheels against stable dependencies (triggered as scheduled jobs),
- building, testing, and publishing release wheels to pypi.org (triggered by commits pushed to the `X.Y.Z` branch on the main repo + manually triggered upload github action workflow),

Ideally this doc should only describe the high level objectives of the CI infrastructure and point the readers to the config files and the `build_tools` folder for the details (which typically evolve faster than we update the doc).

We should carefully use inline comments in those config files and scripts to make sure that people easily understand how they operate and how to conduct common maintenance operations (e.g. rotating secret tokens).

This doc should also consolidate the list of commit message trigger keywords (currently present in the contributors guide if I am not mistaken).

I think the CI doc deserves its own page in our usage guide and should be cross-linked from the contributors and the maintainers docs.

COMMENT:
Hi @ogrisel , I'm new to open source. I would like to work on this issue. Could you please guide me ?

Labels: Documentation, Moderate, Build / CI
High level documentation of the CI infrastructure

As originally discussed in https://github.com/scikit-learn/scikit-learn/pull/25562#discussion_r1098396646:

I think it might be helpful to give a high level description of our CI somewhere in the doc, both for new contributors and maintainers. In particular, we should summarize the various missions of our CI:

- building and testing PRs to give feedback to contributors and reviewers prior to merging (triggered by commits pushed to the PR branch on the contributor's fork),
- building the stable and dev doc websites (triggered by commits pushed to the `main` and `X.Y.Z` branches on the main repo respectively),
- building and testing scikit-learn against the development versions of its dependencies (triggered as scheduled jobs),
- building, testing, and publishing nightly wheels against stable dependencies (triggered as scheduled jobs),
- building, testing, and publishing release wheels to pypi.org (triggered by commits pushed to the `X.Y.Z` branch on the main repo + manually triggered upload github action workflow),

Ideally this doc should only describe the high level objectives of the CI infrastructure and point the readers to the config files and the `build_tools` folder for the details (which typically evolve faster than we update the doc).

We should carefully use inline comments in those config files and scripts to make sure that people easily understand how they operate and how to conduct common maintenance operations (e.g. rotating secret tokens).

This doc should also consolidate the list of commit message trigger keywords (currently present in the contributors guide if I am not mistaken).

I think the CI doc deserves its own page in our usage guide and should be cross-linked from the contributors and the maintainers docs.

COMMENT:
@ggold7046 you should start with an issue labeled as a good first issue. This is best left for one of the more experienced developers on the repo.

Labels: Documentation, Moderate, Build / CI
High level documentation of the CI infrastructure

As originally discussed in https://github.com/scikit-learn/scikit-learn/pull/25562#discussion_r1098396646:

I think it might be helpful to give a high level description of our CI somewhere in the doc, both for new contributors and maintainers. In particular, we should summarize the various missions of our CI:

- building and testing PRs to give feedback to contributors and reviewers prior to merging (triggered by commits pushed to the PR branch on the contributor's fork),
- building the stable and dev doc websites (triggered by commits pushed to the `main` and `X.Y.Z` branches on the main repo respectively),
- building and testing scikit-learn against the development versions of its dependencies (triggered as scheduled jobs),
- building, testing, and publishing nightly wheels against stable dependencies (triggered as scheduled jobs),
- building, testing, and publishing release wheels to pypi.org (triggered by commits pushed to the `X.Y.Z` branch on the main repo + manually triggered upload github action workflow),

Ideally this doc should only describe the high level objectives of the CI infrastructure and point the readers to the config files and the `build_tools` folder for the details (which typically evolve faster than we update the doc).

We should carefully use inline comments in those config files and scripts to make sure that people easily understand how they operate and how to conduct common maintenance operations (e.g. rotating secret tokens).

This doc should also consolidate the list of commit message trigger keywords (currently present in the contributors guide if I am not mistaken).

I think the CI doc deserves its own page in our usage guide and should be cross-linked from the contributors and the maintainers docs.

COMMENT:
Actually I thought this is a document related issue. That's why I wanted to start with it.

Labels: Documentation, Moderate, Build / CI
High level documentation of the CI infrastructure

As originally discussed in https://github.com/scikit-learn/scikit-learn/pull/25562#discussion_r1098396646:

I think it might be helpful to give a high level description of our CI somewhere in the doc, both for new contributors and maintainers. In particular, we should summarize the various missions of our CI:

- building and testing PRs to give feedback to contributors and reviewers prior to merging (triggered by commits pushed to the PR branch on the contributor's fork),
- building the stable and dev doc websites (triggered by commits pushed to the `main` and `X.Y.Z` branches on the main repo respectively),
- building and testing scikit-learn against the development versions of its dependencies (triggered as scheduled jobs),
- building, testing, and publishing nightly wheels against stable dependencies (triggered as scheduled jobs),
- building, testing, and publishing release wheels to pypi.org (triggered by commits pushed to the `X.Y.Z` branch on the main repo + manually triggered upload github action workflow),

Ideally this doc should only describe the high level objectives of the CI infrastructure and point the readers to the config files and the `build_tools` folder for the details (which typically evolve faster than we update the doc).

We should carefully use inline comments in those config files and scripts to make sure that people easily understand how they operate and how to conduct common maintenance operations (e.g. rotating secret tokens).

This doc should also consolidate the list of commit message trigger keywords (currently present in the contributors guide if I am not mistaken).

I think the CI doc deserves its own page in our usage guide and should be cross-linked from the contributors and the maintainers docs.

COMMENT:
@ggold7046 This documentation issue requires understanding our CI infrastructure which has quite a few moving parts now. Although this is a documentation related task, I do not think it is simple.

Labels: Documentation, Moderate, Build / CI
High level documentation of the CI infrastructure

As originally discussed in https://github.com/scikit-learn/scikit-learn/pull/25562#discussion_r1098396646:

I think it might be helpful to give a high level description of our CI somewhere in the doc, both for new contributors and maintainers. In particular, we should summarize the various missions of our CI:

- building and testing PRs to give feedback to contributors and reviewers prior to merging (triggered by commits pushed to the PR branch on the contributor's fork),
- building the stable and dev doc websites (triggered by commits pushed to the `main` and `X.Y.Z` branches on the main repo respectively),
- building and testing scikit-learn against the development versions of its dependencies (triggered as scheduled jobs),
- building, testing, and publishing nightly wheels against stable dependencies (triggered as scheduled jobs),
- building, testing, and publishing release wheels to pypi.org (triggered by commits pushed to the `X.Y.Z` branch on the main repo + manually triggered upload github action workflow),

Ideally this doc should only describe the high level objectives of the CI infrastructure and point the readers to the config files and the `build_tools` folder for the details (which typically evolve faster than we update the doc).

We should carefully use inline comments in those config files and scripts to make sure that people easily understand how they operate and how to conduct common maintenance operations (e.g. rotating secret tokens).

This doc should also consolidate the list of commit message trigger keywords (currently present in the contributors guide if I am not mistaken).

I think the CI doc deserves its own page in our usage guide and should be cross-linked from the contributors and the maintainers docs.

COMMENT:
@thomasjpfan could you tell me if there is some beginner level ci issues which I can do ?

Labels: Documentation, Moderate, Build / CI
Streamlining Bug Fix Releases

Reading over https://github.com/scikit-learn/scikit-learn/pull/25457 I wish we had workflow where we can immediately backport fixes to `1.2.X` once the fix is on `main`. This way we do not need to do a big interactive rebase when we release. We would only need to update the authors list and then cut a release directly from `1.2.X`.

I think the only thing stopping us from directly back-porting fixes to `upstream/1.2.X` is that it updates the documentation right away. In that case, I see two ways around this:

1. A new `doc/1.2.X` branch that gets deployed to the docs. `doc/1.2.X` will always be behind `1.2.X`. When we release, sync `doc/1.2.X` up with `1.2.X`.
2. A new `dev/1.2.X` that we backport fixes to. `dev/1.2.X` will always be ahead of `1.2.X`. When we release, sync `1.2.X` up with `dev/1.2.X`.

I know this adds a little more process, but I think making it easier to cut bug fix releases.

COMMENT:
Option (2) sounds like it would require less (new) infrastructure changes.

A side comment related to backporting: https://meeseeksbox.github.io/ (https://github.com/MeeseeksBox) is used by a few projects that do backports and quite popular. It removes the need to manual backporting and can even act based on merged PR labels.

ps. the title has a typo, should be "Streamlining"

Labels: workflow
Streamlining Bug Fix Releases

Reading over https://github.com/scikit-learn/scikit-learn/pull/25457 I wish we had workflow where we can immediately backport fixes to `1.2.X` once the fix is on `main`. This way we do not need to do a big interactive rebase when we release. We would only need to update the authors list and then cut a release directly from `1.2.X`.

I think the only thing stopping us from directly back-porting fixes to `upstream/1.2.X` is that it updates the documentation right away. In that case, I see two ways around this:

1. A new `doc/1.2.X` branch that gets deployed to the docs. `doc/1.2.X` will always be behind `1.2.X`. When we release, sync `doc/1.2.X` up with `1.2.X`.
2. A new `dev/1.2.X` that we backport fixes to. `dev/1.2.X` will always be ahead of `1.2.X`. When we release, sync `1.2.X` up with `dev/1.2.X`.

I know this adds a little more process, but I think making it easier to cut bug fix releases.

COMMENT:
For `meeseeksbox`, it would require a more significant shift on how we are branching if I recall correctly (not that I am against it). My only concern is about the difficulty of automatically backporting with the way that we currently handle our changelog. We usually get some conflict there which could be problematic with automatic backport but I am not sure. I think that we should be investigating this part.

Regarding the process, we just move some burden that is currently on the release manager to all maintainers (again I am not against it :)). Then, we would need a more systematic way of knowing exactly what goes into the bugfix release as well.

Labels: workflow
Streamlining Bug Fix Releases

Reading over https://github.com/scikit-learn/scikit-learn/pull/25457 I wish we had workflow where we can immediately backport fixes to `1.2.X` once the fix is on `main`. This way we do not need to do a big interactive rebase when we release. We would only need to update the authors list and then cut a release directly from `1.2.X`.

I think the only thing stopping us from directly back-porting fixes to `upstream/1.2.X` is that it updates the documentation right away. In that case, I see two ways around this:

1. A new `doc/1.2.X` branch that gets deployed to the docs. `doc/1.2.X` will always be behind `1.2.X`. When we release, sync `doc/1.2.X` up with `1.2.X`.
2. A new `dev/1.2.X` that we backport fixes to. `dev/1.2.X` will always be ahead of `1.2.X`. When we release, sync `1.2.X` up with `dev/1.2.X`.

I know this adds a little more process, but I think making it easier to cut bug fix releases.

COMMENT:
As for having a bot to do auto backports, there are many GitHub actions to choose from that [trigger on labels](https://github.com/marketplace?type=actions&query=backport+).

As for conflicts on the changelog, I think we can do those manually, right after the PR is merged into `main`. Historically, we typically have around 15 new changelog items for bug fix releases, which feels manageable. Many of the other backported commits are documentation changes that can more easily be automated.

Overall, I want to make it easier for the release manager. 😅

Labels: workflow
Streamlining Bug Fix Releases

Reading over https://github.com/scikit-learn/scikit-learn/pull/25457 I wish we had workflow where we can immediately backport fixes to `1.2.X` once the fix is on `main`. This way we do not need to do a big interactive rebase when we release. We would only need to update the authors list and then cut a release directly from `1.2.X`.

I think the only thing stopping us from directly back-porting fixes to `upstream/1.2.X` is that it updates the documentation right away. In that case, I see two ways around this:

1. A new `doc/1.2.X` branch that gets deployed to the docs. `doc/1.2.X` will always be behind `1.2.X`. When we release, sync `doc/1.2.X` up with `1.2.X`.
2. A new `dev/1.2.X` that we backport fixes to. `dev/1.2.X` will always be ahead of `1.2.X`. When we release, sync `1.2.X` up with `dev/1.2.X`.

I know this adds a little more process, but I think making it easier to cut bug fix releases.

COMMENT:
> As for having a bot to do auto backports, there are many GitHub actions to choose from that [trigger on labels](https://github.com/marketplace?type=actions&query=backport+).

The reason I brought up meseeks specifically is that it is used by other scientific python projects for a while and the developer(s) are from the community. This increases the chances that it fits how projects work (and lower chance of getting compromised?), changes can be made, etc compared to picking a random tool from the GH marketplace.

We could find out how ipython and scikit-image do their changelog to learn about dealing with conflicts.

Spreading the load sounds like a good idea. Anything to make "making a release" less of an event and more of a routine "nothing to see here" task.

Labels: workflow
Streamlining Bug Fix Releases

Reading over https://github.com/scikit-learn/scikit-learn/pull/25457 I wish we had workflow where we can immediately backport fixes to `1.2.X` once the fix is on `main`. This way we do not need to do a big interactive rebase when we release. We would only need to update the authors list and then cut a release directly from `1.2.X`.

I think the only thing stopping us from directly back-porting fixes to `upstream/1.2.X` is that it updates the documentation right away. In that case, I see two ways around this:

1. A new `doc/1.2.X` branch that gets deployed to the docs. `doc/1.2.X` will always be behind `1.2.X`. When we release, sync `doc/1.2.X` up with `1.2.X`.
2. A new `dev/1.2.X` that we backport fixes to. `dev/1.2.X` will always be ahead of `1.2.X`. When we release, sync `1.2.X` up with `dev/1.2.X`.

I know this adds a little more process, but I think making it easier to cut bug fix releases.

COMMENT:
I have seen `meeseeksbox` work well for other libraries. For me, it adds a dependency on another external Heroku service. For a GitHub Action workflow, we can limit the permissions to only allow it to create a PR. (No write permissions to the branches, etc.) In any case, I am okay with any solution we choose.

> We could find out how ipython and scikit-image do their changelog to learn about dealing with conflicts.

Many other open source projects have a folder of change log files, one file per PR. During release time, they run a script to merge the files together. For example, `matplotlib` has a [next_whats_new folder](https://github.com/matplotlib/matplotlib/tree/main/doc/users/next_whats_new) for upcoming changes and a [prev_whats_new folder](https://github.com/matplotlib/matplotlib/tree/main/doc/users/prev_whats_new) with the combined change logs.

Labels: workflow