Talip7/scikit-learn-issues-multilabel huggingface dataset:
Catching deprecation warnings from examples

Up-to-now, we never check deprecation warnings that are raised when executing our examples.
I assume that we could scrap the RST generated file to find such warning:

```python
from pathlib import Path
from pprint import pprint

from sklearn.utils.fixes import VisibleDeprecationWarning

from conf import sphinx_gallery_conf

warning_types = (DeprecationWarning, FutureWarning, VisibleDeprecationWarning)

auto_examples_path = Path(sphinx_gallery_conf["gallery_dirs"][0])

example_raising_warnings = []
for example_filename in auto_examples_path.rglob("*.rst"):
    with open(example_filename, "r") as f:
        lines = f.readlines()
        for line in lines:
            if any([warning_type.__name__ in line for warning_type in warning_types]):
                example_raising_warnings.append(
                    f"{str(example_filename)}: {line.strip()}"
                )

pprint(example_raising_warnings, width=1_000)
```

What I am wondering is exactly how do we want to use it with the CI. Supposedly, it should be only CircleCI but should target only PRs?

COMMENT:
We run those whenever the PR is merged as a part of the CI in `main` though.

Labels: Build / CI
Catching deprecation warnings from examples

Up-to-now, we never check deprecation warnings that are raised when executing our examples.
I assume that we could scrap the RST generated file to find such warning:

```python
from pathlib import Path
from pprint import pprint

from sklearn.utils.fixes import VisibleDeprecationWarning

from conf import sphinx_gallery_conf

warning_types = (DeprecationWarning, FutureWarning, VisibleDeprecationWarning)

auto_examples_path = Path(sphinx_gallery_conf["gallery_dirs"][0])

example_raising_warnings = []
for example_filename in auto_examples_path.rglob("*.rst"):
    with open(example_filename, "r") as f:
        lines = f.readlines()
        for line in lines:
            if any([warning_type.__name__ in line for warning_type in warning_types]):
                example_raising_warnings.append(
                    f"{str(example_filename)}: {line.strip()}"
                )

pprint(example_raising_warnings, width=1_000)
```

What I am wondering is exactly how do we want to use it with the CI. Supposedly, it should be only CircleCI but should target only PRs?

COMMENT:
Looking quickly at examples with warnings in https://github.com/scikit-learn/scikit-learn/issues/27430#issuecomment-1729444562, it seems like you need to special case the release highlights, since we probably don't care too much that old release highlights have warnings. 

Also I feel there may be some annoying situations where we will have to make the code in the example more complicated to be without warnings for the doc build and still run with the doc-min-depencies build, but maybe this is OK.

Labels: Build / CI
Catching deprecation warnings from examples

Up-to-now, we never check deprecation warnings that are raised when executing our examples.
I assume that we could scrap the RST generated file to find such warning:

```python
from pathlib import Path
from pprint import pprint

from sklearn.utils.fixes import VisibleDeprecationWarning

from conf import sphinx_gallery_conf

warning_types = (DeprecationWarning, FutureWarning, VisibleDeprecationWarning)

auto_examples_path = Path(sphinx_gallery_conf["gallery_dirs"][0])

example_raising_warnings = []
for example_filename in auto_examples_path.rglob("*.rst"):
    with open(example_filename, "r") as f:
        lines = f.readlines()
        for line in lines:
            if any([warning_type.__name__ in line for warning_type in warning_types]):
                example_raising_warnings.append(
                    f"{str(example_filename)}: {line.strip()}"
                )

pprint(example_raising_warnings, width=1_000)
```

What I am wondering is exactly how do we want to use it with the CI. Supposedly, it should be only CircleCI but should target only PRs?

COMMENT:
we probably don't care about warnings in doc-min-dependencies?

Labels: Build / CI
Catching deprecation warnings from examples

Up-to-now, we never check deprecation warnings that are raised when executing our examples.
I assume that we could scrap the RST generated file to find such warning:

```python
from pathlib import Path
from pprint import pprint

from sklearn.utils.fixes import VisibleDeprecationWarning

from conf import sphinx_gallery_conf

warning_types = (DeprecationWarning, FutureWarning, VisibleDeprecationWarning)

auto_examples_path = Path(sphinx_gallery_conf["gallery_dirs"][0])

example_raising_warnings = []
for example_filename in auto_examples_path.rglob("*.rst"):
    with open(example_filename, "r") as f:
        lines = f.readlines()
        for line in lines:
            if any([warning_type.__name__ in line for warning_type in warning_types]):
                example_raising_warnings.append(
                    f"{str(example_filename)}: {line.strip()}"
                )

pprint(example_raising_warnings, width=1_000)
```

What I am wondering is exactly how do we want to use it with the CI. Supposedly, it should be only CircleCI but should target only PRs?

COMMENT:
> it seems like you need to special case the release highlights, since we probably don't care too much that old release highlights have warnings.

Yep this one I am not sure about. Up to today, we always tried to still make it work with a comment if required.

> Also I feel there may be some annoying situations where we will have to make the code in the example more complicated to be without warnings for the doc build and still run with the doc-min-depencies build, but maybe this is OK.

It will be case to case I assume. I could at this time to evaluate if we need to bump matplotlib for the mimimum doc to use the simplify code version. Having the lock file is indeed pretty handy to not savage all PRs.

Labels: Build / CI
Catching deprecation warnings from examples

Up-to-now, we never check deprecation warnings that are raised when executing our examples.
I assume that we could scrap the RST generated file to find such warning:

```python
from pathlib import Path
from pprint import pprint

from sklearn.utils.fixes import VisibleDeprecationWarning

from conf import sphinx_gallery_conf

warning_types = (DeprecationWarning, FutureWarning, VisibleDeprecationWarning)

auto_examples_path = Path(sphinx_gallery_conf["gallery_dirs"][0])

example_raising_warnings = []
for example_filename in auto_examples_path.rglob("*.rst"):
    with open(example_filename, "r") as f:
        lines = f.readlines()
        for line in lines:
            if any([warning_type.__name__ in line for warning_type in warning_types]):
                example_raising_warnings.append(
                    f"{str(example_filename)}: {line.strip()}"
                )

pprint(example_raising_warnings, width=1_000)
```

What I am wondering is exactly how do we want to use it with the CI. Supposedly, it should be only CircleCI but should target only PRs?

COMMENT:
> we probably don't care about warnings in doc-min-dependencies?

From experience, this is more that you need to use a new matplotlib functionality to make the new doc pass, and it will break the min doc CI. And then this is a big split to make it work between both. So to me, this is usually coming back to evaluate a bump to the matplotlib version for the min doc.

On the general comment that we "only" get a warning with old version, then I agree with you: I don't really care.

Labels: Build / CI
PERF Regression in HistGradientBoostingClassifier with cython 3

Related to https://github.com/scikit-learn/scikit-learn/issues/27086 but specialized to the regression observed in `HistGradientBoostingClassifier` for better tracking.

With cython 3, it looks that the speed of 2 parts has changed.
- the initial binning is faster, which is a good thing
- the computation of the histograms at each iteration is slower which is a bad thing.

Here's a profiling that clearly shows that the method `HistogramBuilder.compute_histograms_brute` is a lot slower than it used to be

**cython 0.29.36**
![prof_hgbt_cy2](https://github.com/scikit-learn/scikit-learn/assets/34657725/c2785724-52fc-4060-8a16-ec0a57e4f46c)

**cython 3.0.2 master**
![prof_hgbt_cy3](https://github.com/scikit-learn/scikit-learn/assets/34657725/9828cf43-7859-4653-a2bd-f618d9422326)

Looking at the cython annotated version of this function I can see that the generated code is different in both cases (~50% more lines of code with cython 3) but I've no clue how to interpret it. Below is the annotated version just for the `def` part of the following method
![Screenshot 2023-09-20 at 16-12-06 Cython histogram pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/da1495fd-fe7a-40de-b591-2d58549b2121)

**cython 0.29.36**
![Screenshot 2023-09-20 at 16-07-50 Cython histogram 2 pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/989c80dd-95f9-40e1-b8d7-efa3d69bc96a)

**cython 3.0.2 master**
![Screenshot 2023-09-20 at 16-08-47 Cython histogram 3 pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/1dd6b58c-3444-4d70-855e-04cb58ee333c)

I did not find any difference in the body of the method.

Ping @da-woods in case you have some time to take a look :)

COMMENT:
This is a "very bad" thing as computing histograms is often the perf bottleneck with more than 50% of time spend for it.

Labels: Performance, Regression
PERF Regression in HistGradientBoostingClassifier with cython 3

Related to https://github.com/scikit-learn/scikit-learn/issues/27086 but specialized to the regression observed in `HistGradientBoostingClassifier` for better tracking.

With cython 3, it looks that the speed of 2 parts has changed.
- the initial binning is faster, which is a good thing
- the computation of the histograms at each iteration is slower which is a bad thing.

Here's a profiling that clearly shows that the method `HistogramBuilder.compute_histograms_brute` is a lot slower than it used to be

**cython 0.29.36**
![prof_hgbt_cy2](https://github.com/scikit-learn/scikit-learn/assets/34657725/c2785724-52fc-4060-8a16-ec0a57e4f46c)

**cython 3.0.2 master**
![prof_hgbt_cy3](https://github.com/scikit-learn/scikit-learn/assets/34657725/9828cf43-7859-4653-a2bd-f618d9422326)

Looking at the cython annotated version of this function I can see that the generated code is different in both cases (~50% more lines of code with cython 3) but I've no clue how to interpret it. Below is the annotated version just for the `def` part of the following method
![Screenshot 2023-09-20 at 16-12-06 Cython histogram pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/da1495fd-fe7a-40de-b591-2d58549b2121)

**cython 0.29.36**
![Screenshot 2023-09-20 at 16-07-50 Cython histogram 2 pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/989c80dd-95f9-40e1-b8d7-efa3d69bc96a)

**cython 3.0.2 master**
![Screenshot 2023-09-20 at 16-08-47 Cython histogram 3 pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/1dd6b58c-3444-4d70-855e-04cb58ee333c)

I did not find any difference in the body of the method.

Ping @da-woods in case you have some time to take a look :)

COMMENT:
My guess (without yet having had a look at the generated code) would be that this is probably in `_compute_histogram_brute_single_feature` and would be fixed in https://github.com/cython/cython/pull/5685. There will mostly likely be a pointless acquire/release of the GIL around the refnanny initialization of that function.

I'll check that properly later though

Labels: Performance, Regression
PERF Regression in HistGradientBoostingClassifier with cython 3

Related to https://github.com/scikit-learn/scikit-learn/issues/27086 but specialized to the regression observed in `HistGradientBoostingClassifier` for better tracking.

With cython 3, it looks that the speed of 2 parts has changed.
- the initial binning is faster, which is a good thing
- the computation of the histograms at each iteration is slower which is a bad thing.

Here's a profiling that clearly shows that the method `HistogramBuilder.compute_histograms_brute` is a lot slower than it used to be

**cython 0.29.36**
![prof_hgbt_cy2](https://github.com/scikit-learn/scikit-learn/assets/34657725/c2785724-52fc-4060-8a16-ec0a57e4f46c)

**cython 3.0.2 master**
![prof_hgbt_cy3](https://github.com/scikit-learn/scikit-learn/assets/34657725/9828cf43-7859-4653-a2bd-f618d9422326)

Looking at the cython annotated version of this function I can see that the generated code is different in both cases (~50% more lines of code with cython 3) but I've no clue how to interpret it. Below is the annotated version just for the `def` part of the following method
![Screenshot 2023-09-20 at 16-12-06 Cython histogram pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/da1495fd-fe7a-40de-b591-2d58549b2121)

**cython 0.29.36**
![Screenshot 2023-09-20 at 16-07-50 Cython histogram 2 pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/989c80dd-95f9-40e1-b8d7-efa3d69bc96a)

**cython 3.0.2 master**
![Screenshot 2023-09-20 at 16-08-47 Cython histogram 3 pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/1dd6b58c-3444-4d70-855e-04cb58ee333c)

I did not find any difference in the body of the method.

Ping @da-woods in case you have some time to take a look :)

COMMENT:
Unfortunately it doesn't look like what I thought so I'll have to have a bit more of a dig into it if possible.

What should I run if I want to run a histogram myself?

Labels: Performance, Regression
PERF Regression in HistGradientBoostingClassifier with cython 3

Related to https://github.com/scikit-learn/scikit-learn/issues/27086 but specialized to the regression observed in `HistGradientBoostingClassifier` for better tracking.

With cython 3, it looks that the speed of 2 parts has changed.
- the initial binning is faster, which is a good thing
- the computation of the histograms at each iteration is slower which is a bad thing.

Here's a profiling that clearly shows that the method `HistogramBuilder.compute_histograms_brute` is a lot slower than it used to be

**cython 0.29.36**
![prof_hgbt_cy2](https://github.com/scikit-learn/scikit-learn/assets/34657725/c2785724-52fc-4060-8a16-ec0a57e4f46c)

**cython 3.0.2 master**
![prof_hgbt_cy3](https://github.com/scikit-learn/scikit-learn/assets/34657725/9828cf43-7859-4653-a2bd-f618d9422326)

Looking at the cython annotated version of this function I can see that the generated code is different in both cases (~50% more lines of code with cython 3) but I've no clue how to interpret it. Below is the annotated version just for the `def` part of the following method
![Screenshot 2023-09-20 at 16-12-06 Cython histogram pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/da1495fd-fe7a-40de-b591-2d58549b2121)

**cython 0.29.36**
![Screenshot 2023-09-20 at 16-07-50 Cython histogram 2 pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/989c80dd-95f9-40e1-b8d7-efa3d69bc96a)

**cython 3.0.2 master**
![Screenshot 2023-09-20 at 16-08-47 Cython histogram 3 pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/1dd6b58c-3444-4d70-855e-04cb58ee333c)

I did not find any difference in the body of the method.

Ping @da-woods in case you have some time to take a look :)

COMMENT:
I looked through the C code images and couldn't find anything that could make a difference. The different number of lines is partly due to slightly safer memoryview refcounting and mostly preprocessor adaptations for PyPy and the Limited C-API. Nothing to worry about here, I'd say.

A C level profile would help, e.g. using Linux perf. Anyone up for that?

Labels: Performance, Regression
PERF Regression in HistGradientBoostingClassifier with cython 3

Related to https://github.com/scikit-learn/scikit-learn/issues/27086 but specialized to the regression observed in `HistGradientBoostingClassifier` for better tracking.

With cython 3, it looks that the speed of 2 parts has changed.
- the initial binning is faster, which is a good thing
- the computation of the histograms at each iteration is slower which is a bad thing.

Here's a profiling that clearly shows that the method `HistogramBuilder.compute_histograms_brute` is a lot slower than it used to be

**cython 0.29.36**
![prof_hgbt_cy2](https://github.com/scikit-learn/scikit-learn/assets/34657725/c2785724-52fc-4060-8a16-ec0a57e4f46c)

**cython 3.0.2 master**
![prof_hgbt_cy3](https://github.com/scikit-learn/scikit-learn/assets/34657725/9828cf43-7859-4653-a2bd-f618d9422326)

Looking at the cython annotated version of this function I can see that the generated code is different in both cases (~50% more lines of code with cython 3) but I've no clue how to interpret it. Below is the annotated version just for the `def` part of the following method
![Screenshot 2023-09-20 at 16-12-06 Cython histogram pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/da1495fd-fe7a-40de-b591-2d58549b2121)

**cython 0.29.36**
![Screenshot 2023-09-20 at 16-07-50 Cython histogram 2 pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/989c80dd-95f9-40e1-b8d7-efa3d69bc96a)

**cython 3.0.2 master**
![Screenshot 2023-09-20 at 16-08-47 Cython histogram 3 pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/1dd6b58c-3444-4d70-855e-04cb58ee333c)

I did not find any difference in the body of the method.

Ping @da-woods in case you have some time to take a look :)

COMMENT:
> A C level profile would help, e.g. using Linux perf. Anyone up for that?

I've done that before and would be happy to have a look at that if I can get some instructions about what to run on the scikit-learn side

Labels: Performance, Regression
PERF Regression in HistGradientBoostingClassifier with cython 3

Related to https://github.com/scikit-learn/scikit-learn/issues/27086 but specialized to the regression observed in `HistGradientBoostingClassifier` for better tracking.

With cython 3, it looks that the speed of 2 parts has changed.
- the initial binning is faster, which is a good thing
- the computation of the histograms at each iteration is slower which is a bad thing.

Here's a profiling that clearly shows that the method `HistogramBuilder.compute_histograms_brute` is a lot slower than it used to be

**cython 0.29.36**
![prof_hgbt_cy2](https://github.com/scikit-learn/scikit-learn/assets/34657725/c2785724-52fc-4060-8a16-ec0a57e4f46c)

**cython 3.0.2 master**
![prof_hgbt_cy3](https://github.com/scikit-learn/scikit-learn/assets/34657725/9828cf43-7859-4653-a2bd-f618d9422326)

Looking at the cython annotated version of this function I can see that the generated code is different in both cases (~50% more lines of code with cython 3) but I've no clue how to interpret it. Below is the annotated version just for the `def` part of the following method
![Screenshot 2023-09-20 at 16-12-06 Cython histogram pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/da1495fd-fe7a-40de-b591-2d58549b2121)

**cython 0.29.36**
![Screenshot 2023-09-20 at 16-07-50 Cython histogram 2 pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/989c80dd-95f9-40e1-b8d7-efa3d69bc96a)

**cython 3.0.2 master**
![Screenshot 2023-09-20 at 16-08-47 Cython histogram 3 pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/1dd6b58c-3444-4d70-855e-04cb58ee333c)

I did not find any difference in the body of the method.

Ping @da-woods in case you have some time to take a look :)

COMMENT:
You can best find how to call histograms in the tests, eg https://github.com/scikit-learn/scikit-learn/blob/ba7d86956da03aa4fd230b1bbe8df57b63cf2dc0/sklearn/ensemble/_hist_gradient_boosting/tests/test_histogram.py#L21.

Gradients and hessians are arrays of the same length as the data (numbers of observations or n_samples), often within 10k - 10M. For benchmarking, the values don’t really matter (any random sampling is fine).

Note that `_build_histogram_naive` is only used for testing, so irrelevant for us.

The function that is actually called is `compute_histograms_brute`. Examples on how to construct a `HistogramBuilder` and call this method are in https://github.com/scikit-learn/scikit-learn/blob/ba7d86956da03aa4fd230b1bbe8df57b63cf2dc0/sklearn/ensemble/_hist_gradient_boosting/tests/test_splitting.py#L900.

@da-woods Do you need more than that?

Labels: Performance, Regression
PERF Regression in HistGradientBoostingClassifier with cython 3

Related to https://github.com/scikit-learn/scikit-learn/issues/27086 but specialized to the regression observed in `HistGradientBoostingClassifier` for better tracking.

With cython 3, it looks that the speed of 2 parts has changed.
- the initial binning is faster, which is a good thing
- the computation of the histograms at each iteration is slower which is a bad thing.

Here's a profiling that clearly shows that the method `HistogramBuilder.compute_histograms_brute` is a lot slower than it used to be

**cython 0.29.36**
![prof_hgbt_cy2](https://github.com/scikit-learn/scikit-learn/assets/34657725/c2785724-52fc-4060-8a16-ec0a57e4f46c)

**cython 3.0.2 master**
![prof_hgbt_cy3](https://github.com/scikit-learn/scikit-learn/assets/34657725/9828cf43-7859-4653-a2bd-f618d9422326)

Looking at the cython annotated version of this function I can see that the generated code is different in both cases (~50% more lines of code with cython 3) but I've no clue how to interpret it. Below is the annotated version just for the `def` part of the following method
![Screenshot 2023-09-20 at 16-12-06 Cython histogram pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/da1495fd-fe7a-40de-b591-2d58549b2121)

**cython 0.29.36**
![Screenshot 2023-09-20 at 16-07-50 Cython histogram 2 pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/989c80dd-95f9-40e1-b8d7-efa3d69bc96a)

**cython 3.0.2 master**
![Screenshot 2023-09-20 at 16-08-47 Cython histogram 3 pyx](https://github.com/scikit-learn/scikit-learn/assets/34657725/1dd6b58c-3444-4d70-855e-04cb58ee333c)

I did not find any difference in the body of the method.

Ping @da-woods in case you have some time to take a look :)

COMMENT:
That sounds fairly straight forward thanks

Labels: Performance, Regression