Talip7/scikit-learn-issues-multilabel huggingface dataset:
Upgrade free-threading CI to run with pytest-freethreaded instead of pytest-xdist

There is a new kid on the block that should help us find out whether scikit-learn and its dependencies can be reliably considered free-threading compatible:

https://pypi.org/project/pytest-freethreaded/

Let's try to adopt it in scikit-learn.

Here is a possible plan:

- first run the tests locally a few times and see if they are tests (or a set of interacting tests) that cause a crash or a failure, open an issue for each of them, possibly upstream and then mark them as skipped under free-threading builds with a reference to the issue in the "reason" field;
- then upgrade our nightly free-threading scheduled CI run to use `pytest-freethreaded`.

Any comments @lesteve @jeremiedbb @ngoldbaum?

EDIT: anyone interested in getting hands on the first item can find this resource useful:

https://py-free-threading.github.io/

EDIT 2: there is also the [pytest-run-parallel](https://github.com/Quansight-Labs/pytest-run-parallel) plugin that can serve a similar purpose.

EDIT3: here is a TODO/plan for this problem:



- [x] resync and simplify #30041 to leverage https://github.com/Quansight-Labs/pytest-run-parallel/pull/19, https://github.com/Quansight-Labs/pytest-run-parallel/pull/33 and https://github.com/Quansight-Labs/pytest-run-parallel/pull/34
- [x] configure the `[free-threading]` CI to run with `pytest-run-parallel` #32023
- [x] enable the free-threading flag in Cython extension #31342
  - [x] investigate and fix crashes and test failures or open tracking issues and mark tests as `thread_unsafe` ;)
- [ ] benchmark the use of threading, in particular when we we expect nested parallelism between Python threads (e.g. in `GridSearchCV` with the "threading" backend of joblib) and BLAS or OpenMP native threading in the underlying estimators.
- [ ] communicate results on a blog post / pydata presentation / social media.

COMMENT:
Also, I just created https://github.com/Quansight-Labs/free-threaded-compatibility/issues/92 to mention these issues around writing thread-safe tests in the guide.

Labels: Build / CI, Needs Decision, module:test-suite, free-threading
Upgrade free-threading CI to run with pytest-freethreaded instead of pytest-xdist

There is a new kid on the block that should help us find out whether scikit-learn and its dependencies can be reliably considered free-threading compatible:

https://pypi.org/project/pytest-freethreaded/

Let's try to adopt it in scikit-learn.

Here is a possible plan:

- first run the tests locally a few times and see if they are tests (or a set of interacting tests) that cause a crash or a failure, open an issue for each of them, possibly upstream and then mark them as skipped under free-threading builds with a reference to the issue in the "reason" field;
- then upgrade our nightly free-threading scheduled CI run to use `pytest-freethreaded`.

Any comments @lesteve @jeremiedbb @ngoldbaum?

EDIT: anyone interested in getting hands on the first item can find this resource useful:

https://py-free-threading.github.io/

EDIT 2: there is also the [pytest-run-parallel](https://github.com/Quansight-Labs/pytest-run-parallel) plugin that can serve a similar purpose.

EDIT3: here is a TODO/plan for this problem:



- [x] resync and simplify #30041 to leverage https://github.com/Quansight-Labs/pytest-run-parallel/pull/19, https://github.com/Quansight-Labs/pytest-run-parallel/pull/33 and https://github.com/Quansight-Labs/pytest-run-parallel/pull/34
- [x] configure the `[free-threading]` CI to run with `pytest-run-parallel` #32023
- [x] enable the free-threading flag in Cython extension #31342
  - [x] investigate and fix crashes and test failures or open tracking issues and mark tests as `thread_unsafe` ;)
- [ ] benchmark the use of threading, in particular when we we expect nested parallelism between Python threads (e.g. in `GridSearchCV` with the "threading" backend of joblib) and BLAS or OpenMP native threading in the underlying estimators.
- [ ] communicate results on a blog post / pydata presentation / social media.

COMMENT:
> Since you're running the same test in many threads and are implicitly using global state (the file system), then of course that's not thread safe.

But `tmpdir` and `tmp_path` are precisely in charge of creating a unique temp folder for your test. When you use them with `pytest-xdist`, there is no concurrency problem when using them naively, even if all Python worker processes managed by `pytest-xdist` share the same file-system: each test sees only its own temp folder. The problem here is that their implicit usage contract is broken if their initialization is shared for several tests running in concurrent threads.

Labels: Build / CI, Needs Decision, module:test-suite, free-threading
Upgrade free-threading CI to run with pytest-freethreaded instead of pytest-xdist

There is a new kid on the block that should help us find out whether scikit-learn and its dependencies can be reliably considered free-threading compatible:

https://pypi.org/project/pytest-freethreaded/

Let's try to adopt it in scikit-learn.

Here is a possible plan:

- first run the tests locally a few times and see if they are tests (or a set of interacting tests) that cause a crash or a failure, open an issue for each of them, possibly upstream and then mark them as skipped under free-threading builds with a reference to the issue in the "reason" field;
- then upgrade our nightly free-threading scheduled CI run to use `pytest-freethreaded`.

Any comments @lesteve @jeremiedbb @ngoldbaum?

EDIT: anyone interested in getting hands on the first item can find this resource useful:

https://py-free-threading.github.io/

EDIT 2: there is also the [pytest-run-parallel](https://github.com/Quansight-Labs/pytest-run-parallel) plugin that can serve a similar purpose.

EDIT3: here is a TODO/plan for this problem:



- [x] resync and simplify #30041 to leverage https://github.com/Quansight-Labs/pytest-run-parallel/pull/19, https://github.com/Quansight-Labs/pytest-run-parallel/pull/33 and https://github.com/Quansight-Labs/pytest-run-parallel/pull/34
- [x] configure the `[free-threading]` CI to run with `pytest-run-parallel` #32023
- [x] enable the free-threading flag in Cython extension #31342
  - [x] investigate and fix crashes and test failures or open tracking issues and mark tests as `thread_unsafe` ;)
- [ ] benchmark the use of threading, in particular when we we expect nested parallelism between Python threads (e.g. in `GridSearchCV` with the "threading" backend of joblib) and BLAS or OpenMP native threading in the underlying estimators.
- [ ] communicate results on a blog post / pydata presentation / social media.

COMMENT:
> When you use them with pytest-xdist, there is no concurrency problem when using them naively, even if all Python worker processes managed by pytest-xdist share the same file-system: each test sees only its own temp folder.

I want to double-check this, since I think if you were running the same test many times in parallel with xdist you'd have the same issue.

Labels: Build / CI, Needs Decision, module:test-suite, free-threading
Upgrade free-threading CI to run with pytest-freethreaded instead of pytest-xdist

There is a new kid on the block that should help us find out whether scikit-learn and its dependencies can be reliably considered free-threading compatible:

https://pypi.org/project/pytest-freethreaded/

Let's try to adopt it in scikit-learn.

Here is a possible plan:

- first run the tests locally a few times and see if they are tests (or a set of interacting tests) that cause a crash or a failure, open an issue for each of them, possibly upstream and then mark them as skipped under free-threading builds with a reference to the issue in the "reason" field;
- then upgrade our nightly free-threading scheduled CI run to use `pytest-freethreaded`.

Any comments @lesteve @jeremiedbb @ngoldbaum?

EDIT: anyone interested in getting hands on the first item can find this resource useful:

https://py-free-threading.github.io/

EDIT 2: there is also the [pytest-run-parallel](https://github.com/Quansight-Labs/pytest-run-parallel) plugin that can serve a similar purpose.

EDIT3: here is a TODO/plan for this problem:



- [x] resync and simplify #30041 to leverage https://github.com/Quansight-Labs/pytest-run-parallel/pull/19, https://github.com/Quansight-Labs/pytest-run-parallel/pull/33 and https://github.com/Quansight-Labs/pytest-run-parallel/pull/34
- [x] configure the `[free-threading]` CI to run with `pytest-run-parallel` #32023
- [x] enable the free-threading flag in Cython extension #31342
  - [x] investigate and fix crashes and test failures or open tracking issues and mark tests as `thread_unsafe` ;)
- [ ] benchmark the use of threading, in particular when we we expect nested parallelism between Python threads (e.g. in `GridSearchCV` with the "threading" backend of joblib) and BLAS or OpenMP native threading in the underlying estimators.
- [ ] communicate results on a blog post / pydata presentation / social media.

COMMENT:
As @ngoldbaum mentioned, tests that use `tmpdir` should append an extra UUID per thread, so that each file does not get overwritten/unexpectedly removed by other thread. 

This goes in line with the philosophy of being aware of concurrent computing, as opposed to having a complex and opaque machinery that tries to abstract the problem away.

I understand that going through each test in the library takes a lot of time, but doing so will ensure that each test function is thread-safe (even if the solution sometimes implies adding locks) and it will future-proof it for future contributors, as adding a CI (for example), will force them to not use global state patterns.

It is difficult to get away from thinking in global state, which is common in single-threaded imperative or object oriented computing. But free-threaded deals with a different paradigm, and requires different abstractions and patterns

Labels: Build / CI, Needs Decision, module:test-suite, free-threading
TransformedTargetRegressor with Pipeline is not fitting model upon calling .fit

### Describe the bug

A common use case is to use Pipeline to transform the feature set and to wrap it with TransformedTargetRegressor to transform the response variable. But when used in combination and calling fit on the TransformedTargetRegressor object, the model internal to the Pipeline is not actually fit. 

### Steps/Code to Reproduce

```python
from sklearn.preprocessing import StandardScaler
from sklearn_pandas import DataFrameMapper
from xgboost import XGBRegressor
from sklearn.compose import TransformedTargetRegressor
from sklearn import datasets
from sklearn.pipeline import Pipeline
import pandas as pd
import numpy as np

iris = datasets.load_iris()
df = pd.DataFrame(iris.data, columns=iris.feature_names)
y_col = "sepal length (cm)"
x_cols = [x for x in iris.feature_names if x != y_col]

model = XGBRegressor()

mod_pipeline = Pipeline([("scaler", StandardScaler()), ("model", model)])

mod_pipeline = TransformedTargetRegressor(regressor=mod_pipeline, func=np.log1p, inverse_func=np.expm1)

mod_pipeline.fit(df[x_cols], df[y_col])

mod_pipeline.regressor['model'].__sklearn_is_fitted__()
```

### Expected Results

`True`

### Actual Results

`False`

### Versions

```shell
System:
    python: 3.11.9 (main, May 17 2024, 12:31:23) [Clang 14.0.3 (clang-1403.0.22.14.1)]
executable: /Users/jmaddalena/projects/vesta/.venv/bin/python
   machine: macOS-14.5-x86_64-i386-64bit

Python dependencies:
      sklearn: 1.4.1.post1
          pip: 24.0
   setuptools: 66.1.1
        numpy: 1.26.4
        scipy: 1.12.0
       Cython: 3.0.9
       pandas: 2.2.2
   matplotlib: 3.8.3
       joblib: 1.3.2
threadpoolctl: 3.3.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 11
         prefix: libomp
       filepath: /Users/jmaddalena/projects/vesta/.venv/lib/python3.11/site-packages/sklearn/.dylibs/libomp.dylib
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 11
         prefix: libopenblas
       filepath: /Users/jmaddalena/projects/vesta/.venv/lib/python3.11/site-packages/numpy/.dylibs/libopenblas64_.0.dylib
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: Nehalem

       user_api: blas
   internal_api: openblas
    num_threads: 11
         prefix: libopenblas
       filepath: /Users/jmaddalena/projects/vesta/.venv/lib/python3.11/site-packages/scipy/.dylibs/libopenblas.0.dylib
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: Nehalem
```

COMMENT:
This expected. You are accessing the unfitted estimator. The fitted one is available via the `regressor_` attribute:

```python
mod_pipeline.regressor_
```

Labels: Bug, Needs Triage
TransformedTargetRegressor with Pipeline is not fitting model upon calling .fit

### Describe the bug

A common use case is to use Pipeline to transform the feature set and to wrap it with TransformedTargetRegressor to transform the response variable. But when used in combination and calling fit on the TransformedTargetRegressor object, the model internal to the Pipeline is not actually fit. 

### Steps/Code to Reproduce

```python
from sklearn.preprocessing import StandardScaler
from sklearn_pandas import DataFrameMapper
from xgboost import XGBRegressor
from sklearn.compose import TransformedTargetRegressor
from sklearn import datasets
from sklearn.pipeline import Pipeline
import pandas as pd
import numpy as np

iris = datasets.load_iris()
df = pd.DataFrame(iris.data, columns=iris.feature_names)
y_col = "sepal length (cm)"
x_cols = [x for x in iris.feature_names if x != y_col]

model = XGBRegressor()

mod_pipeline = Pipeline([("scaler", StandardScaler()), ("model", model)])

mod_pipeline = TransformedTargetRegressor(regressor=mod_pipeline, func=np.log1p, inverse_func=np.expm1)

mod_pipeline.fit(df[x_cols], df[y_col])

mod_pipeline.regressor['model'].__sklearn_is_fitted__()
```

### Expected Results

`True`

### Actual Results

`False`

### Versions

```shell
System:
    python: 3.11.9 (main, May 17 2024, 12:31:23) [Clang 14.0.3 (clang-1403.0.22.14.1)]
executable: /Users/jmaddalena/projects/vesta/.venv/bin/python
   machine: macOS-14.5-x86_64-i386-64bit

Python dependencies:
      sklearn: 1.4.1.post1
          pip: 24.0
   setuptools: 66.1.1
        numpy: 1.26.4
        scipy: 1.12.0
       Cython: 3.0.9
       pandas: 2.2.2
   matplotlib: 3.8.3
       joblib: 1.3.2
threadpoolctl: 3.3.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 11
         prefix: libomp
       filepath: /Users/jmaddalena/projects/vesta/.venv/lib/python3.11/site-packages/sklearn/.dylibs/libomp.dylib
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 11
         prefix: libopenblas
       filepath: /Users/jmaddalena/projects/vesta/.venv/lib/python3.11/site-packages/numpy/.dylibs/libopenblas64_.0.dylib
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: Nehalem

       user_api: blas
   internal_api: openblas
    num_threads: 11
         prefix: libopenblas
       filepath: /Users/jmaddalena/projects/vesta/.venv/lib/python3.11/site-packages/scipy/.dylibs/libopenblas.0.dylib
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: Nehalem
```

COMMENT:
Thank you, that is very helpful! May I ask why there is both an unfitted estimator (`mod_pipeline.regressor`) and a fitted one (`mod_pipeline.regressor_`) when there is not an equivalent in the Pipeline object? I imagine I'm not the only user confused by this.

Labels: Bug, Needs Triage
DOC rework the example presenting the regularization path of Lasso, Lasso-LARS, and Elastic Net

We recently merge two examples and the resulting example is shown here: https://scikit-learn.org/dev/auto_examples/linear_model/plot_lasso_lasso_lars_elasticnet_path.html

This example should be revisited where we should have more narrative in a tutorial-like style. Indeed, this example could explain in more details what is a regularization path and discuss the difference between Lasso and Lasso-LARS, and between Lasso and ElasticNet.

Some of the experiment are really closed to the one presented in this paper: https://hastie.su.domains/Papers/LARS/LeastAngle_2002.pdf

COMMENT:
I'd like to work on this issue, if it's available.

Labels: Documentation
DOC rework the example presenting the regularization path of Lasso, Lasso-LARS, and Elastic Net

We recently merge two examples and the resulting example is shown here: https://scikit-learn.org/dev/auto_examples/linear_model/plot_lasso_lasso_lars_elasticnet_path.html

This example should be revisited where we should have more narrative in a tutorial-like style. Indeed, this example could explain in more details what is a regularization path and discuss the difference between Lasso and Lasso-LARS, and between Lasso and ElasticNet.

Some of the experiment are really closed to the one presented in this paper: https://hastie.su.domains/Papers/LARS/LeastAngle_2002.pdf

COMMENT:
@glemaitre I would like to ask for clarification regarding tutorial-like style. Do we have to explain the examples, like, explain how lasso -LARS works differently from lasso and elastic net regression through the example?

Labels: Documentation
DOC rework the example presenting the regularization path of Lasso, Lasso-LARS, and Elastic Net

We recently merge two examples and the resulting example is shown here: https://scikit-learn.org/dev/auto_examples/linear_model/plot_lasso_lasso_lars_elasticnet_path.html

This example should be revisited where we should have more narrative in a tutorial-like style. Indeed, this example could explain in more details what is a regularization path and discuss the difference between Lasso and Lasso-LARS, and between Lasso and ElasticNet.

Some of the experiment are really closed to the one presented in this paper: https://hastie.su.domains/Papers/LARS/LeastAngle_2002.pdf

COMMENT:
> Do we have to explain the examples, like, explain how lasso -LARS works differently from lasso and elastic net regression through the example?

Kind of. Basically, since we want to show differences through those plots, I think that we need to:

- first, introduce what is a regularization path
- then, explain the difference between Lasso and ElasticNet regarding the regularization and look at the path because we will have something along the same line
- finally explain the difference between Lasso and Lasso-LARS, that is related to the solver, and how it impacts the path also.

Labels: Documentation
DOC rework the example presenting the regularization path of Lasso, Lasso-LARS, and Elastic Net

We recently merge two examples and the resulting example is shown here: https://scikit-learn.org/dev/auto_examples/linear_model/plot_lasso_lasso_lars_elasticnet_path.html

This example should be revisited where we should have more narrative in a tutorial-like style. Indeed, this example could explain in more details what is a regularization path and discuss the difference between Lasso and Lasso-LARS, and between Lasso and ElasticNet.

Some of the experiment are really closed to the one presented in this paper: https://hastie.su.domains/Papers/LARS/LeastAngle_2002.pdf

COMMENT:
hi I'd like to pick this up, anything specific I should keep in mind apart from the discussion taken up already above?

cc @StefanieSenger

Labels: Documentation
DOC rework the example presenting the regularization path of Lasso, Lasso-LARS, and Elastic Net

We recently merge two examples and the resulting example is shown here: https://scikit-learn.org/dev/auto_examples/linear_model/plot_lasso_lasso_lars_elasticnet_path.html

This example should be revisited where we should have more narrative in a tutorial-like style. Indeed, this example could explain in more details what is a regularization path and discuss the difference between Lasso and Lasso-LARS, and between Lasso and ElasticNet.

Some of the experiment are really closed to the one presented in this paper: https://hastie.su.domains/Papers/LARS/LeastAngle_2002.pdf

COMMENT:
@virchan is already working on this here: #30028

Labels: Documentation
DOC merging the examples related to OPTICS, DBSCAN, and HDBSCAN

As stated in https://github.com/scikit-learn/scikit-learn/issues/27151, it would be great to reduce the number of examples in the gallery.

Right now, we have three examples for:

- OPTICS: https://scikit-learn.org/dev/auto_examples/cluster/plot_optics.html#sphx-glr-auto-examples-cluster-plot-optics-py
- DBSCAN: https://scikit-learn.org/dev/auto_examples/cluster/plot_dbscan.html#sphx-glr-auto-examples-cluster-plot-dbscan-py
- HDBSCAN: https://scikit-learn.org/dev/auto_examples/cluster/plot_hdbscan.html#sphx-glr-auto-examples-cluster-plot-hdbscan-py

Those clustering methods are really close to each others; some being an improvement from another one. Therefore, we could rework a single example that is not only a demo but rather show the pros & cons from each approach.

COMMENT:
/take

Labels: Documentation, help wanted
DOC merging the examples related to OPTICS, DBSCAN, and HDBSCAN

As stated in https://github.com/scikit-learn/scikit-learn/issues/27151, it would be great to reduce the number of examples in the gallery.

Right now, we have three examples for:

- OPTICS: https://scikit-learn.org/dev/auto_examples/cluster/plot_optics.html#sphx-glr-auto-examples-cluster-plot-optics-py
- DBSCAN: https://scikit-learn.org/dev/auto_examples/cluster/plot_dbscan.html#sphx-glr-auto-examples-cluster-plot-dbscan-py
- HDBSCAN: https://scikit-learn.org/dev/auto_examples/cluster/plot_hdbscan.html#sphx-glr-auto-examples-cluster-plot-hdbscan-py

Those clustering methods are really close to each others; some being an improvement from another one. Therefore, we could rework a single example that is not only a demo but rather show the pros & cons from each approach.

COMMENT:
@saldanhad are you working on this? I'd be happy to help!

Labels: Documentation, help wanted
DOC merging the examples related to OPTICS, DBSCAN, and HDBSCAN

As stated in https://github.com/scikit-learn/scikit-learn/issues/27151, it would be great to reduce the number of examples in the gallery.

Right now, we have three examples for:

- OPTICS: https://scikit-learn.org/dev/auto_examples/cluster/plot_optics.html#sphx-glr-auto-examples-cluster-plot-optics-py
- DBSCAN: https://scikit-learn.org/dev/auto_examples/cluster/plot_dbscan.html#sphx-glr-auto-examples-cluster-plot-dbscan-py
- HDBSCAN: https://scikit-learn.org/dev/auto_examples/cluster/plot_hdbscan.html#sphx-glr-auto-examples-cluster-plot-hdbscan-py

Those clustering methods are really close to each others; some being an improvement from another one. Therefore, we could rework a single example that is not only a demo but rather show the pros & cons from each approach.

COMMENT:
may i take this one? would love to try to streamline the examples here.

Labels: Documentation, help wanted
DOC merging the examples related to OPTICS, DBSCAN, and HDBSCAN

As stated in https://github.com/scikit-learn/scikit-learn/issues/27151, it would be great to reduce the number of examples in the gallery.

Right now, we have three examples for:

- OPTICS: https://scikit-learn.org/dev/auto_examples/cluster/plot_optics.html#sphx-glr-auto-examples-cluster-plot-optics-py
- DBSCAN: https://scikit-learn.org/dev/auto_examples/cluster/plot_dbscan.html#sphx-glr-auto-examples-cluster-plot-dbscan-py
- HDBSCAN: https://scikit-learn.org/dev/auto_examples/cluster/plot_hdbscan.html#sphx-glr-auto-examples-cluster-plot-hdbscan-py

Those clustering methods are really close to each others; some being an improvement from another one. Therefore, we could rework a single example that is not only a demo but rather show the pros & cons from each approach.

COMMENT:
There's been some work in https://github.com/scikit-learn/scikit-learn/pull/31102, but that PR is now closed. Future contributors should look at that PR and the corresponding reviews.

Labels: Documentation, help wanted