Weekly CI run with NVidia GPU hardware

Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)

COMMENT:
I agree that we will need a mechanism that allows core maintainers to approve a run of this custom runner. Labelling sounds good. I've also seen people use comments. Not sure what the pros and cons are.

As I understand it in the PyTorch case the attackers managed to run arbitrary code that happened to be "set this machine up to be a runner for our own repo". With a manual approval process (different from PyTorch) you'd have to spot that the PR contains such a "install backdoor" bit of code. Which can be easy but also hard. Once the backdoor is installed, all bets are off/there is nothing we can do anymore. In the PyTorch case they used their backdoor to wait for other workflows with more privileges and triggered by other people to run. Then grabbed the tokens/secrets those workflows had and used them to "do crime".

I think some things we could consider:
* the VM a workflow runs on should be deleted at the end of each run. That way an installed backdoor only survives for as long as that particular run.
* a short-ish timeout for workflow runs. That way `GITHUB_TOKEN`s you steal from a running workflow become useless after a short amount of time. The timeout has to be enforced "from the outside" as well as not be editable from inside the runner or our repo (so must be configured somewhere inside cirun's UI)
* as normal user or `root` on the VM that a workflow is running on I should not be able to access the cloud account that "owns" the VM. There are things like "metadata service" and "machine tokens" and so on that give you some information/access just based on the fact that you are a VM owned by the account. Otherwise people could try to replace the image of the VM that the runner uses, with one that includes a backdoor.
* disallow almost all egress from the VM. I think there is little need for our workflows to access the internet
* preventing people from running bitcoin miners is a bit trickier. There are things like the techniques used by https://github.com/cryptnono/cryptnono and such that are in use on mybinder.org - I'd hope we can piggyback on something from the cloud provider?
* minimal privileges everywhere for everything (standard measure)
* :+1: Thomas' plan to not use the workflow config of a PR when running the workflows of a PR, but sticking to the one in `main`
* as a general aim: make it so that you can't reconfigure the thing you are in, from inside it. What I mean is "you shouldn't be able to reconfigure the VM (image) from inside the VM" or "you should not be able to configure the workflow config of your PR from inside the PR"

---

Something I don't quite grasp yet is why custom runners are much more dangerous/susceptible than the GH provided runners. If we can understand this difference we will probably get some good ideas for how to harden our custom runner.

Labels: Build / CI, help wanted, Array API