Talip7/scikit-learn-issues-multilabel huggingface dataset:
Weekly CI run with NVidia GPU hardware

Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)

COMMENT:
@aktech thanks for adding a security section in the cirun.io doc! Maybe you want to add a link to the github doc you mentioned to show that you are aware of this and/or for people who are curious.

Labels: Build / CI, help wanted, Array API
Weekly CI run with NVidia GPU hardware

Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)

COMMENT:
@lesteve [done](https://docs.cirun.io/security), thanks for pointing that out.

Labels: Build / CI, help wanted, Array API
Weekly CI run with NVidia GPU hardware

Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)

COMMENT:
The labeling idea sounds about right (although I am clearly not an expert). Another thing I was wondering about: is there a way to make sure the secrets like the one to upload on PyPI or anaconda.org scientific-python-nightly-wheels are not available on the custom runner?

Edit: maybe https://github.blog/2021-04-13-implementing-least-privilege-for-secrets-in-github-actions/

Labels: Build / CI, help wanted, Array API
Weekly CI run with NVidia GPU hardware

Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)

COMMENT:
For PyPI, we do not use secrets anymore and use [Trusted Publisher](https://docs.pypi.org/trusted-publishers/)

As for uploads to `scientific-python-nightly-wheels` or the staging index, we should create an GitHub Actions environment for them.

Labels: Build / CI, help wanted, Array API
Weekly CI run with NVidia GPU hardware

Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)

COMMENT:
I opened https://github.com/scikit-learn/scikit-learn/pull/28441 to add an environment for the `upload_anaconda` secrets.

Labels: Build / CI, help wanted, Array API
Weekly CI run with NVidia GPU hardware

Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)

COMMENT:
> After a few more iterations, it works now! https://github.com/thomasjpfan/scikit-learn/actions/runs/7820229387/job/21334434689

> Next step is to to wrap it up into our workflow for triggering cirun.

Now that the security concerns seem to have been cleared, I would like to understand in more details what is required to move forward.

From reading @thomasjpfan config at https://github.com/scikit-learn/scikit-learn/compare/main...thomasjpfan:scikit-learn:cirun_gcp, I have the following questions:

- where does the `third-campus-393023:scikit-learn-gpu-v4` machine image come from?

- do we need the `.devcontainer/devcontainer.json` as part of the CI itself, or was it just for local dev?

- in the previous plan written in https://github.com/scikit-learn/scikit-learn/issues/24491#issuecomment-1934123952, you mentioned using `workflow_dispatch` but I do not see that in the linked diff. Is the use of `workflow_dispatch` still relevant? or is the `pull_request` dispatch with `types: [labeled]` enough? From a security point of view, it seems that the `workflow_dispatch` is still required to avoid allowing external contributors to cheat by tweaking the github workflow files in their own PR to trigger the GPU spin-up without explicit maintainers control.

- assuming the latter and that we configure the `if: ${{ github.event.label.name == 'gpu' }}` condition on a github actions job configured with `runs-on: "cirun-gpu-runner--${{ github.run_id }}"`, and the surrounding workflow is configured with:

  ```python
  on:
    pull_request:
      types: [labeled]
  ```

  will cirun spin-up a costly GPU VM just to run the build condition (e.g. check the presence of the label on the PR) or will the GPU VM start (and be build) only if the trigger condition evaluates to true?

Labels: Build / CI, help wanted, Array API
Weekly CI run with NVidia GPU hardware

Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)

COMMENT:
I can answer a couple of questions:

> where does the third-campus-393023:scikit-learn-gpu-v4 machine image come from?

This is the custom image @thomasjpfan created in his GCP account with nvidia drivers installed (and maybe something else too).

>will cirun spin-up a costly GPU VM just to run the build condition (e.g. check the presence of the label on the PR) or will the GPU VM start (and be build) only if the trigger condition evaluates to true?

You can do this by checking if GPU tests should run or not in GitHub hosted runner (non-gpu) and then trigger the job with GPU if required, example:

<img width="654" alt="Screenshot 2024-03-15 at 1 42 52 pm" src="https://github.com/scikit-learn/scikit-learn/assets/5647941/a7bc0853-e5c8-4240-8ca0-bcabeb196e88">

Ref: https://github.com/scverse/anndata/actions/runs/8277985942

Labels: Build / CI, help wanted, Array API
Weekly CI run with NVidia GPU hardware

Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)

COMMENT:
> where does the third-campus-393023:scikit-learn-gpu-v4 machine image come from?

This is a custom image on my GCP account with conda + nvidia drivers installed. This makes start up times much faster, at the cost of storing an image in GCP.

> do we need the .devcontainer/devcontainer.json as part of the CI itself, or was it just for local dev?

I mixed up testing Codespaces into my GPU+CI testing branch.

> Is the use of workflow_dispatch still relevant? 

This was my mistake, the GitHub event should either be `workflow_run` or `pull_request_target`. I think these events are still required because of the security concerns. I feel like `pull_request_target` could be good enough, but that needs some testing. I did not configured this in my experimentation.

CIRun does have [access controls](https://docs.cirun.io/reference/access-control), but I do not think it works for our workflow of PRs from external users. 
Also, we'll need


Labels: Build / CI, help wanted, Array API
Weekly CI run with NVidia GPU hardware

Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)

COMMENT:
> Also, we'll need

It seems that you did not finish your sentence.

Labels: Build / CI, help wanted, Array API
Weekly CI run with NVidia GPU hardware

Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)

COMMENT:
> It seems that you did not finish your sentence.

I reordered the paragraphs and did not remove the last part in the editing. 😅

Labels: Build / CI, help wanted, Array API
Weekly CI run with NVidia GPU hardware

Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)

COMMENT:
>> where does the third-campus-393023:scikit-learn-gpu-v4 machine image come from?

> This is a custom image on my GCP account with conda + nvidia drivers installed. This makes start up times much faster, at the cost of storing an image in GCP.

Could you please push the `Dockerfile` to your branch to make it more convenient to update this container from time to time when dependencies are updated?

Labels: Build / CI, help wanted, Array API
Weekly CI run with NVidia GPU hardware

Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)

COMMENT:
I did not use a `Dockerfile` to create the custom image on GCP. I had a running instance, installed the Nvidia drivers + conda, and then created a clone.

I do not know if there is a way to create a custom image with Docker on GCP.

REF: https://docs.cirun.io/custom-images/cloud-custom-images#gcp-custom-images

Labels: Build / CI, help wanted, Array API
Weekly CI run with NVidia GPU hardware

Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)

COMMENT:
I don't think it's possible to generate VM images from Dockerfile, the last I checked. To programmatically generate VM images you can use something like packer: https://developer.hashicorp.com/packer/integrations/hashicorp/googlecompute/latest/components/builder/googlecompute

Example: https://github.com/actions/runner-images/blob/98d2bcc93e055d8892f9446e72a4da66b334bfb1/images/ubuntu/templates/ubuntu-22.04.pkr.hcl

Labels: Build / CI, help wanted, Array API
Weekly CI run with NVidia GPU hardware

Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)

COMMENT:
Why would a custom VM image be a better solution that using a custom docker image in a standard GCP image?

Labels: Build / CI, help wanted, Array API
Weekly CI run with NVidia GPU hardware

Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)

COMMENT:
> Why would a custom VM image be a better solution that using a custom docker image in a standard GCP image?

It's not about which one is better, either can be done with different levels of complexity. You can build a custom docker image, but in that case you'll have to do these things:

- Still have to create a custom image with [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installing-the-nvidia-container-toolkit) installed, this will help in building and running GPU-accelerated containers.
- Run your tests inside docker, which will add more complexity in the scikit-learn CI.

Labels: Build / CI, help wanted, Array API
Weekly CI run with NVidia GPU hardware

Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)

COMMENT:
For the record, GitHub has recently announced a public beta for GitHub-hosted runners with GPU capabilities:

- https://github.blog/2024-04-02-bringing-enterprise-level-security-and-even-more-power-to-github-hosted-runners/#gpu-hosted-runners-available-in-public-beta-%f0%9f%8e%89

However, the scikit-learn organization does not seem to have the ability to create such runners with its GitHub Team plan at the moment.

EDIT: I wrote the above too quickly, apparently @betatim figured it out: we need to use a "partner image" from NVIDIA when creating the runner to see the GPU option.

Labels: Build / CI, help wanted, Array API
add **fit_params to sklearn.compose.ColumnTransformer().fit()

### Describe the workflow you want to enable

The `fit` function of both [sklearn.pipeline](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.pipeline).Pipeline and [sklearn.pipeline](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.pipeline).FeatureUnion supports `**fit_params`: 
```python
def fit(X, y=None, **fit_params):
    ...
```
Citing the documentation:
>`**fit_params : dict of string -> object
>    Parameters passed to the fit method of each step, where each parameter name is prefixed such that parameter p for step s has key s__p.


Conversely, [sklearn.compose](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.compose).ColumnTransformer does not support `fit_params`:

```python
def fit(X, y=None):
    ...
```

As far as I'm aware, it is therefore not possible to pass fit parameters to the individual column transforms.


### Describe your proposed solution

The fit function of [sklearn.compose](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.compose).ColumnTransformer should support `fit_params`, using the same logic as [sklearn.pipeline](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.pipeline).Pipeline:

```python
def fit(X, y=None, **fit_params):
    ...
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
https://github.com/scikit-learn/scikit-learn/pull/24027 fixes this.

Labels: New Feature, module:compose
DOC Mention pandas dataframe support in `ColumnTransformer` in FAQ

### Describe the issue linked to the documentation

FAQ question: [Why does Scikit-learn not directly work with, for example, pandas.DataFrame?](https://scikit-learn.org/stable/faq.html#why-does-scikit-learn-not-directly-work-with-for-example-pandas-dataframe) - I think it would be worthwhile to mention pandas dataframe support in [`ColumnTransformer`](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html#sklearn.compose.ColumnTransformer) (even though estimators do no directly work with dataframes).

### Suggest a potential alternative/fix

As above

COMMENT:
I agree this is a good suggestion.

Labels: Documentation
DOC Mention pandas dataframe support in `ColumnTransformer` in FAQ

### Describe the issue linked to the documentation

FAQ question: [Why does Scikit-learn not directly work with, for example, pandas.DataFrame?](https://scikit-learn.org/stable/faq.html#why-does-scikit-learn-not-directly-work-with-for-example-pandas-dataframe) - I think it would be worthwhile to mention pandas dataframe support in [`ColumnTransformer`](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html#sklearn.compose.ColumnTransformer) (even though estimators do no directly work with dataframes).

### Suggest a potential alternative/fix

As above

COMMENT:
I am new to Open Source can I work on the issue?

Labels: Documentation
DOC Mention pandas dataframe support in `ColumnTransformer` in FAQ

### Describe the issue linked to the documentation

FAQ question: [Why does Scikit-learn not directly work with, for example, pandas.DataFrame?](https://scikit-learn.org/stable/faq.html#why-does-scikit-learn-not-directly-work-with-for-example-pandas-dataframe) - I think it would be worthwhile to mention pandas dataframe support in [`ColumnTransformer`](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html#sklearn.compose.ColumnTransformer) (even though estimators do no directly work with dataframes).

### Suggest a potential alternative/fix

As above

COMMENT:
Feel free to open a PR for this issue. See the contributing guide for help: https://scikit-learn.org/dev/developers/contributing.html#documentation

Labels: Documentation
DOC Mention pandas dataframe support in `ColumnTransformer` in FAQ

### Describe the issue linked to the documentation

FAQ question: [Why does Scikit-learn not directly work with, for example, pandas.DataFrame?](https://scikit-learn.org/stable/faq.html#why-does-scikit-learn-not-directly-work-with-for-example-pandas-dataframe) - I think it would be worthwhile to mention pandas dataframe support in [`ColumnTransformer`](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html#sklearn.compose.ColumnTransformer) (even though estimators do no directly work with dataframes).

### Suggest a potential alternative/fix

As above

COMMENT:
Closing as fixed by #24532.

Labels: Documentation
Implement p-value splitting criterion for Decision Trees

### Describe the workflow you want to enable

The current list of valid criterions for Decision Trees are:

{“squared_error”, “friedman_mse”, “absolute_error”, “poisson”}

With regard to regression problems, I have run into numerous situations where I would very much like to make splits on the data based on a t-test statistics comparing the means between the two split subsamples.

The primary issue with using MSE or Friedman MSE is that the quality of a split is dependent on both the marginal impact of splitting on a particularly variable, as well as the frequency in which that split is triggered. It's not uncommon where I want to identify splits that have highly different means, even though that splitting rule would come into effect rarely.

See this post (I am not the author) for more information on why splitting on p-values could be valuable: http://erikerlandson.github.io/blog/2016/05/26/measuring-decision-tree-split-quality-with-test-statistic-p-values/

### Describe your proposed solution

The solution would be to add a new splitting criterion called "p-value" or "t-test" or something similar. The quality of a split would be measured based on either the p-value or the magnitude of the t-statistic.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
It seems an elegant way to give awareness of samples size during the decision to split or not. I am curious to know how it works on several tasks. Someone has more info (bench, papers) ? And/or intention to implement it ?

Labels: New Feature, module:tree, Needs Decision - Include Feature