Repository: scikit-learn/scikit-learn
Issue category: Feature Request

Issue:
Weekly CI run with NVidia GPU hardware
Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)


Hints:
Other opensource projects like dask and numba have accounts on https://gpuci.gpuopenanalytics.com/ for instance. Not sure what is the procedure to get scikit-learn accepted there (even with a limited quota of a few GPU hours per month).
Ive started looking into this
As discussed during the triaging meeting, we could use https://docs.cirun.io/ which is free for open source projects.

We would then need to register a credit card for our numfocus account on an AWS or Google Cloud.

If we do a few weekly runs, that should be cheap enough.

If we want to make it possible to trigger a GPU run from a git commit message in a PR we need to make sure that cirun would not spawn a GPU instance just to skip the build if the flag is not present in the git commit message, but it might be doable.
Progress update as of a few weeks ago: I got `cirun.io` working on my fork, but I ran into an issue with it stalling with PyTorch. I suspect it was because I was using GPUs that were too old, but I ran out of time to debug.

To reduce cost:

- I had GitHub Actions build the wheel and ship it over to `cirun.io`.
- I had a prebuilt image with the cuda dependencies installed.
Great news! Thanks very much for the progress report @thomasjpfan.
Is there a way to help out/join in? I think having a CI sooner rather than later would be really useful.

I think cirun is our only option
@thomasjpfan do you want someone else to try to takeover from your work in https://github.com/thomasjpfan/scikit-learn/tree/cirun_gpu if you don't have time to finalize this work yourself?
The latest work is at https://github.com/thomasjpfan/scikit-learn/tree/cirun_gcp (GCP)

I'll take another took at it this weekend. If I do not have a PR by Monday, then let's schedule a call so I can do a handoff.
After a few more iterations, it works now! https://github.com/thomasjpfan/scikit-learn/actions/runs/7820229387/job/21334434689

Next step is to to wrap it up into our workflow for triggering cirun.
It also works with T4s: https://github.com/thomasjpfan/scikit-learn/actions/runs/7820372944/job/21334925253 it looks to spin up faster, because there are more T4s available. 
My understanding is that using custom Github runners exposes you to a lot of security issues, I am curious to know if there is any thoughts on how to mitigate this in scikit-learn, e.g. by only allowing a scheduled weekly run or some other way that make sure that running the GPU workflow is fully controlled by maintainers.

For exemple, according to [Hardening for self-hosted runners](https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#hardening-for-self-hosted-runners) "self-hosted runners should almost never be used for public repositories".

Also using a custom github runner was a key aspect in the [PyTorch suppply-chain article](https://simonwillison.net/2024/Jan/14/supply-chain-attack-on-pytorch/)

I had a quick look at https://cirun.io and I could not find any mention about security there ...


I'm +1 on at least having a weekly run.

The hard part comes from PRs. I proposed using a "run-gpu-ci" GitHub label to trigger the CI. Here are the details:

1. When a maintainer adds a "run-gpu-ci" label, it will trigger a wheel build on the PR itself.
2. The wheel is send over a "run on gpu" workflow using `workflow_dispatch`.
	- This workflow type runs in the context of the main branch, so the only adjustable input is the wheel itself.
	- With `workflow_dispatch` only the "cirun.io" configuration on the `main` branch is used. PRs can not configure the runner itself.
4. The "run gpu" removes the "run-gpu-ci" label and sends the wheel to `cirun` for testing. (`pytest --pyargs sklearn`)

With the above approach, it requires a maintainer to add a "run-gpu-ci" label each time they want to trigger a gpu ci run. In the end, it is using labels to implement "Require approval for all outside collaborators’" just for the GPU runner.

There are two security concerns:

1. We make sure that only the scikit-learn team can add the "run-gpu-ci" label.
2. Before adding the label, the reviewer needs to make sure the code does not do anything malicious. Specifically, one can still run arbitrary code in a test.

I say 2 is the bigger concern, because we can miss something. @aktech What are the security concerns with running arbitrary code using `cirun.io`?

I agree that we will need a mechanism that allows core maintainers to approve a run of this custom runner. Labelling sounds good. I've also seen people use comments. Not sure what the pros and cons are.

As I understand it in the PyTorch case the attackers managed to run arbitrary code that happened to be "set this machine up to be a runner for our own repo". With a manual approval process (different from PyTorch) you'd have to spot that the PR contains such a "install backdoor" bit of code. Which can be easy but also hard. Once the backdoor is installed, all bets are off/there is nothing we can do anymore. In the PyTorch case they used their backdoor to wait for other workflows with more privileges and triggered by other people to run. Then grabbed the tokens/secrets those workflows had and used them to "do crime".

I think some things we could consider:
* the VM a workflow runs on should be deleted at the end of each run. That way an installed backdoor only survives for as long as that particular run.
* a short-ish timeout for workflow runs. That way `GITHUB_TOKEN`s you steal from a running workflow become useless after a short amount of time. The timeout has to be enforced "from the outside" as well as not be editable from inside the runner or our repo (so must be configured somewhere inside cirun's UI)
* as normal user or `root` on the VM that a workflow is running on I should not be able to access the cloud account that "owns" the VM. There are things like "metadata service" and "machine tokens" and so on that give you some information/access just based on the fact that you are a VM owned by the account. Otherwise people could try to replace the image of the VM that the runner uses, with one that includes a backdoor.
* disallow almost all egress from the VM. I think there is little need for our workflows to access the internet
* preventing people from running bitcoin miners is a bit trickier. There are things like the techniques used by https://github.com/cryptnono/cryptnono and such that are in use on mybinder.org - I'd hope we can piggyback on something from the cloud provider?
* minimal privileges everywhere for everything (standard measure)
* :+1: Thomas' plan to not use the workflow config of a PR when running the workflows of a PR, but sticking to the one in `main`
* as a general aim: make it so that you can't reconfigure the thing you are in, from inside it. What I mean is "you shouldn't be able to reconfigure the VM (image) from inside the VM" or "you should not be able to configure the workflow config of your PR from inside the PR"

---

Something I don't quite grasp yet is why custom runners are much more dangerous/susceptible than the GH provided runners. If we can understand this difference we will probably get some good ideas for how to harden our custom runner.
Thanks @thomasjpfan for the ping. I am more than happy to address any concerns and help implement CI for sckit-learn.

> @aktech What are the security concerns with running arbitrary code using cirun.io?

There aren't any security concerns that you wouldn't have on GitHub hosted runners. The runners created via cirun are ephemeral and are destroyed right after job completion. The most information one can know is the GCP project name, that's the basic metadata on the VM. The github token for runner provision is destroyed right after it's connected to GitHub and is ephemeral as well.

If you like you can also restrict runner creation by users/teams/permissions in the GitHub orgs via: https://docs.cirun.io/reference/access-control - conda-forge [uses this](https://github.com/conda-forge/.cirun) to restrict usage and access control CI.

> Something I don't quite grasp yet is why custom runners are much more dangerous/susceptible than the GH provided runners. If we can understand this difference we will probably get some good ideas for how to harden our custom runner.

@betatim checkout the second link [2]

As per GitHub docs

> This is not an issue with GitHub-hosted runners because each GitHub-hosted runner is always a clean isolated virtual machine, and it is destroyed at the end of the job execution.

Which is true for isolated VMs created by cirun as well.

Some ref:
- [1] https://docs.cirun.io/security
- [2] https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners#self-hosted-runner-security

@aktech thanks for adding a security section in the cirun.io doc! Maybe you want to add a link to the github doc you mentioned to show that you are aware of this and/or for people who are curious.
@lesteve [done](https://docs.cirun.io/security), thanks for pointing that out.
The labeling idea sounds about right (although I am clearly not an expert). Another thing I was wondering about: is there a way to make sure the secrets like the one to upload on PyPI or anaconda.org scientific-python-nightly-wheels are not available on the custom runner?

Edit: maybe https://github.blog/2021-04-13-implementing-least-privilege-for-secrets-in-github-actions/
For PyPI, we do not use secrets anymore and use [Trusted Publisher](https://docs.pypi.org/trusted-publishers/)

As for uploads to `scientific-python-nightly-wheels` or the staging index, we should create an GitHub Actions environment for them.
I opened https://github.com/scikit-learn/scikit-learn/pull/28441 to add an environment for the `upload_anaconda` secrets.
> After a few more iterations, it works now! https://github.com/thomasjpfan/scikit-learn/actions/runs/7820229387/job/21334434689

> Next step is to to wrap it up into our workflow for triggering cirun.

Now that the security concerns seem to have been cleared, I would like to understand in more details what is required to move forward.

From reading @thomasjpfan config at https://github.com/scikit-learn/scikit-learn/compare/main...thomasjpfan:scikit-learn:cirun_gcp, I have the following questions:

- where does the `third-campus-393023:scikit-learn-gpu-v4` machine image come from?

- do we need the `.devcontainer/devcontainer.json` as part of the CI itself, or was it just for local dev?

- in the previous plan written in https://github.com/scikit-learn/scikit-learn/issues/24491#issuecomment-1934123952, you mentioned using `workflow_dispatch` but I do not see that in the linked diff. Is the use of `workflow_dispatch` still relevant? or is the `pull_request` dispatch with `types: [labeled]` enough? From a security point of view, it seems that the `workflow_dispatch` is still required to avoid allowing external contributors to cheat by tweaking the github workflow files in their own PR to trigger the GPU spin-up without explicit maintainers control.

- assuming the latter and that we configure the `if: ${{ github.event.label.name == 'gpu' }}` condition on a github actions job configured with `runs-on: "cirun-gpu-runner--${{ github.run_id }}"`, and the surrounding workflow is configured with:

  ```python
  on:
    pull_request:
      types: [labeled]
  ```

  will cirun spin-up a costly GPU VM just to run the build condition (e.g. check the presence of the label on the PR) or will the GPU VM start (and be build) only if the trigger condition evaluates to true?

I can answer a couple of questions:

> where does the third-campus-393023:scikit-learn-gpu-v4 machine image come from?

This is the custom image @thomasjpfan created in his GCP account with nvidia drivers installed (and maybe something else too).

>will cirun spin-up a costly GPU VM just to run the build condition (e.g. check the presence of the label on the PR) or will the GPU VM start (and be build) only if the trigger condition evaluates to true?

You can do this by checking if GPU tests should run or not in GitHub hosted runner (non-gpu) and then trigger the job with GPU if required, example:

<img width="654" alt="Screenshot 2024-03-15 at 1 42 52 pm" src="https://github.com/scikit-learn/scikit-learn/assets/5647941/a7bc0853-e5c8-4240-8ca0-bcabeb196e88">

Ref: https://github.com/scverse/anndata/actions/runs/8277985942
> where does the third-campus-393023:scikit-learn-gpu-v4 machine image come from?

This is a custom image on my GCP account with conda + nvidia drivers installed. This makes start up times much faster, at the cost of storing an image in GCP.

> do we need the .devcontainer/devcontainer.json as part of the CI itself, or was it just for local dev?

I mixed up testing Codespaces into my GPU+CI testing branch.

> Is the use of workflow_dispatch still relevant? 

This was my mistake, the GitHub event should either be `workflow_run` or `pull_request_target`. I think these events are still required because of the security concerns. I feel like `pull_request_target` could be good enough, but that needs some testing. I did not configured this in my experimentation.

CIRun does have [access controls](https://docs.cirun.io/reference/access-control), but I do not think it works for our workflow of PRs from external users. 
Also, we'll need



> Also, we'll need

It seems that you did not finish your sentence.
> It seems that you did not finish your sentence.

I reordered the paragraphs and did not remove the last part in the editing. 😅
>> where does the third-campus-393023:scikit-learn-gpu-v4 machine image come from?

> This is a custom image on my GCP account with conda + nvidia drivers installed. This makes start up times much faster, at the cost of storing an image in GCP.

Could you please push the `Dockerfile` to your branch to make it more convenient to update this container from time to time when dependencies are updated?
I did not use a `Dockerfile` to create the custom image on GCP. I had a running instance, installed the Nvidia drivers + conda, and then created a clone.

I do not know if there is a way to create a custom image with Docker on GCP.

REF: https://docs.cirun.io/custom-images/cloud-custom-images#gcp-custom-images
I don't think it's possible to generate VM images from Dockerfile, the last I checked. To programmatically generate VM images you can use something like packer: https://developer.hashicorp.com/packer/integrations/hashicorp/googlecompute/latest/components/builder/googlecompute

Example: https://github.com/actions/runner-images/blob/98d2bcc93e055d8892f9446e72a4da66b334bfb1/images/ubuntu/templates/ubuntu-22.04.pkr.hcl
Why would a custom VM image be a better solution that using a custom docker image in a standard GCP image?
> Why would a custom VM image be a better solution that using a custom docker image in a standard GCP image?

It's not about which one is better, either can be done with different levels of complexity. You can build a custom docker image, but in that case you'll have to do these things:

- Still have to create a custom image with [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installing-the-nvidia-container-toolkit) installed, this will help in building and running GPU-accelerated containers.
- Run your tests inside docker, which will add more complexity in the scikit-learn CI.



For the record, GitHub has recently announced a public beta for GitHub-hosted runners with GPU capabilities:

- https://github.blog/2024-04-02-bringing-enterprise-level-security-and-even-more-power-to-github-hosted-runners/#gpu-hosted-runners-available-in-public-beta-%f0%9f%8e%89

However, the scikit-learn organization does not seem to have the ability to create such runners with its GitHub Team plan at the moment.

EDIT: I wrote the above too quickly, apparently @betatim figured it out: we need to use a "partner image" from NVIDIA when creating the runner to see the GPU option.

Patch:
diff --git a/.github/workflows/cuda-gpu-ci.yml b/.github/workflows/cuda-gpu-ci.yml
new file mode 100644
index 0000000000000..d962145cfbbc7
--- /dev/null
+++ b/.github/workflows/cuda-gpu-ci.yml
@@ -0,0 +1,47 @@
+name: CUDA GPU
+on:
+  workflow_dispatch:
+    inputs:
+      pr_id:
+        description: Test the contents of this Pull Request
+        required: true
+
+permissions: read-all
+
+jobs:
+  tests:
+    runs-on:
+      group: cuda-gpu-runner-group
+    name: Run Array API unit tests
+    steps:
+      - uses: actions/setup-python@v4
+        with:
+          python-version: '3.12'
+      - name: Checkout main repository
+        uses: actions/checkout@v2
+      - name: Checkout a particular Pull Request
+        if: inputs.pr_id
+        env:
+          PR_ID: ${{ inputs.pr_id }}
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          gh pr checkout ${{ env.PR_ID }}
+      - name: Cache conda environment
+        id: cache-conda
+        uses: actions/cache@v3
+        with:
+          path: ~/conda
+          key: ${{ runner.os }}-build-${{ hashFiles('build_tools/github/create_gpu_environment.sh') }}-${{ hashFiles('build_tools/github/pylatest_conda_forge_cuda_array-api_linux-64_conda.lock') }}
+      - name: Install miniforge
+        if: ${{ steps.cache-conda.outputs.cache-hit != 'true' }}
+        run: bash build_tools/github/create_gpu_environment.sh
+      - name: Install scikit-learn
+        run: |
+          source "${HOME}/conda/etc/profile.d/conda.sh"
+          conda activate sklearn
+          pip install --verbose --no-build-isolation --config-settings editable-verbose=true --editable .
+      - name: Run array API tests
+        run: |
+          source "${HOME}/conda/etc/profile.d/conda.sh"
+          conda activate sklearn
+          pytest -k 'array_api'
diff --git a/.github/workflows/update-lock-files.yml b/.github/workflows/update-lock-files.yml
index 8301b45fa37fa..d07f62a0433a8 100644
--- a/.github/workflows/update-lock-files.yml
+++ b/.github/workflows/update-lock-files.yml
@@ -6,6 +6,10 @@ on:
   schedule:
     - cron: '0 5 * * 1'
 
+# XXX Set the right permissions, per step??
+# Can we set read only at the global level here and then elevate to write for some steps?
+#permissions: read-all
+
 jobs:
   update_lock_files:
     if: github.repository == 'scikit-learn/scikit-learn'
@@ -25,6 +29,8 @@ jobs:
           - name: cirrus-arm
             update_script_args: "--select-tag arm"
             additional_commit_message: "[cirrus arm]"
+          - name: array API
+            update_script_args: "--select-tag cuda"
 
     steps:
       - uses: actions/checkout@v4
@@ -56,6 +62,14 @@ jobs:
             ### Note
             If the CI tasks fail, create a new branch based on this PR and add the required fixes to that branch.
 
+      # The CUDA workflow needs to be triggered explicitly as it uses an expensive runner
+      - name: Trigger additional tests
+        if: steps.cpr.outputs.pull-request-number != '' && matrix.name == 'array API'
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          gh workflow run .github/workflows/cuda-gpu-ci.yml -f pr_id=${{steps.cpr.outputs.pull-request-number}}
+
       - name: Check Pull Request
         if: steps.cpr.outputs.pull-request-number != ''
         run: |
diff --git a/build_tools/github/create_gpu_environment.sh b/build_tools/github/create_gpu_environment.sh
new file mode 100644
index 0000000000000..e03fa0691d5b8
--- /dev/null
+++ b/build_tools/github/create_gpu_environment.sh
@@ -0,0 +1,17 @@
+#!/bin/bash
+
+set -e
+set -x
+
+curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
+bash Miniforge3-$(uname)-$(uname -m).sh -b -p "${HOME}/conda"
+source "${HOME}/conda/etc/profile.d/conda.sh"
+
+
+# defines the get_dep and show_installed_libraries functions
+source build_tools/shared.sh
+conda activate base
+
+# XXX switch once https://github.com/scikit-learn/scikit-learn/pull/29176 is merged
+conda install -c conda-forge "$(get_dep conda-lock min)" -y
+conda-lock install --name sklearn build_tools/github/pylatest_conda_forge_cuda_array-api_linux-64_conda.lock
diff --git a/build_tools/update_environments_and_lock_files.py b/build_tools/update_environments_and_lock_files.py
index 92d97709386d1..2f6263cdd961d 100644
--- a/build_tools/update_environments_and_lock_files.py
+++ b/build_tools/update_environments_and_lock_files.py
@@ -90,13 +90,33 @@ def remove_from(alist, to_remove):
 
 
 build_metadata_list = [
+    {
+        "name": "pylatest_conda_forge_cuda_array-api_linux-64",
+        "type": "conda",
+        "tag": "cuda",
+        "folder": "build_tools/github",
+        "platform": "linux-64",
+        "channels": ["conda-forge", "pytorch", "nvidia"],
+        "conda_dependencies": common_dependencies
+        + [
+            "ccache",
+            # Make sure pytorch comes from the pytorch channel and not conda-forge
+            "pytorch::pytorch",
+            "pytorch-cuda",
+            "polars",
+            "pyarrow",
+            "cupy",
+            "array-api-compat",
+            "array-api-strict",
+        ],
+    },
     {
         "name": "pylatest_conda_forge_mkl_linux-64",
         "type": "conda",
         "tag": "main-ci",
         "folder": "build_tools/azure",
         "platform": "linux-64",
-        "channel": "conda-forge",
+        "channels": ["conda-forge"],
         "conda_dependencies": common_dependencies
         + [
             "ccache",
@@ -118,7 +138,7 @@ def remove_from(alist, to_remove):
         "tag": "main-ci",
         "folder": "build_tools/azure",
         "platform": "osx-64",
-        "channel": "conda-forge",
+        "channels": ["conda-forge"],
         "conda_dependencies": common_dependencies
         + [
             "ccache",
@@ -135,7 +155,7 @@ def remove_from(alist, to_remove):
         "tag": "main-ci",
         "folder": "build_tools/azure",
         "platform": "osx-64",
-        "channel": "defaults",
+        "channels": ["defaults"],
         "conda_dependencies": remove_from(
             common_dependencies, ["cython", "threadpoolctl", "meson-python"]
         )
@@ -157,7 +177,7 @@ def remove_from(alist, to_remove):
         "tag": "main-ci",
         "folder": "build_tools/azure",
         "platform": "linux-64",
-        "channel": "defaults",
+        "channels": ["defaults"],
         "conda_dependencies": remove_from(
             common_dependencies,
             ["pandas", "threadpoolctl", "pip", "ninja", "meson-python"],
@@ -183,7 +203,7 @@ def remove_from(alist, to_remove):
         "tag": "main-ci",
         "folder": "build_tools/azure",
         "platform": "linux-64",
-        "channel": "conda-forge",
+        "channels": ["conda-forge"],
         "conda_dependencies": (
             common_dependencies_without_coverage
             + docstring_test_dependencies
@@ -200,7 +220,7 @@ def remove_from(alist, to_remove):
         "tag": "main-ci",
         "folder": "build_tools/azure",
         "platform": "linux-64",
-        "channel": "defaults",
+        "channels": ["defaults"],
         "conda_dependencies": ["python", "ccache"],
         "pip_dependencies": (
             remove_from(common_dependencies, ["python", "blas", "pip"])
@@ -217,7 +237,7 @@ def remove_from(alist, to_remove):
         "tag": "scipy-dev",
         "folder": "build_tools/azure",
         "platform": "linux-64",
-        "channel": "defaults",
+        "channels": ["defaults"],
         "conda_dependencies": ["python", "ccache"],
         "pip_dependencies": (
             remove_from(
@@ -251,7 +271,7 @@ def remove_from(alist, to_remove):
         "tag": "main-ci",
         "folder": "build_tools/azure",
         "platform": "win-64",
-        "channel": "conda-forge",
+        "channels": ["conda-forge"],
         "conda_dependencies": remove_from(common_dependencies, ["pandas", "pyamg"])
         + [
             "wheel",
@@ -268,7 +288,7 @@ def remove_from(alist, to_remove):
         "tag": "main-ci",
         "folder": "build_tools/circle",
         "platform": "linux-64",
-        "channel": "conda-forge",
+        "channels": ["conda-forge"],
         "conda_dependencies": common_dependencies_without_coverage
         + [
             "scikit-image",
@@ -320,7 +340,7 @@ def remove_from(alist, to_remove):
         "tag": "main-ci",
         "folder": "build_tools/circle",
         "platform": "linux-64",
-        "channel": "conda-forge",
+        "channels": ["conda-forge"],
         "conda_dependencies": common_dependencies_without_coverage
         + [
             "scikit-image",
@@ -355,7 +375,7 @@ def remove_from(alist, to_remove):
         "tag": "arm",
         "folder": "build_tools/cirrus",
         "platform": "linux-aarch64",
-        "channel": "conda-forge",
+        "channels": ["conda-forge"],
         "conda_dependencies": remove_from(
             common_dependencies_without_coverage, ["pandas", "pyamg"]
         )
@@ -472,7 +492,9 @@ def get_conda_environment_content(build_metadata):
 # following script to centralize the configuration for CI builds:
 # build_tools/update_environments_and_lock_files.py
 channels:
-  - {{ build_metadata['channel'] }}
+  {% for channel in build_metadata['channels'] %}
+  - {{ channel }}
+  {% endfor %}
 dependencies:
   {% for conda_dep in build_metadata['conda_dependencies'] %}
   - {{ conda_dep | get_package_with_constraint(build_metadata) }}
@@ -720,6 +742,7 @@ def main(select_build, skip_build, select_tag, verbose, very_verbose):
     filtered_conda_build_metadata_list = [
         each for each in filtered_build_metadata_list if each["type"] == "conda"
     ]
+
     if filtered_conda_build_metadata_list:
         logger.info("# Writing conda environments")
         write_all_conda_environments(filtered_conda_build_metadata_list)


Test patch:
diff --git a/build_tools/github/pylatest_conda_forge_cuda_array-api_linux-64_conda.lock b/build_tools/github/pylatest_conda_forge_cuda_array-api_linux-64_conda.lock
new file mode 100644
index 0000000000000..38742e34cb4ea
--- /dev/null
+++ b/build_tools/github/pylatest_conda_forge_cuda_array-api_linux-64_conda.lock
@@ -0,0 +1,263 @@
+# Generated by conda-lock.
+# platform: linux-64
+# input_hash: d227a7296fd0dae4731df4c0b76aa31dbb49785f4cc8f726b511ee9d856fa802
+@EXPLICIT
+https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2#d7c89558ba9fa0495403155b64376d81
+https://conda.anaconda.org/conda-forge/linux-64/ca-certificates-2024.6.2-hbcca054_0.conda#847c3c2905cc467cea52c24f9cfa8080
+https://conda.anaconda.org/conda-forge/noarch/cuda-version-12.1-h1d6eff3_3.conda#913018efd4acd03c48f15cb60d2bbf97
+https://conda.anaconda.org/conda-forge/noarch/font-ttf-dejavu-sans-mono-2.37-hab24e00_0.tar.bz2#0c96522c6bdaed4b1566d11387caaf45
+https://conda.anaconda.org/conda-forge/noarch/font-ttf-inconsolata-3.000-h77eed37_0.tar.bz2#34893075a5c9e55cdafac56607368fc6
+https://conda.anaconda.org/conda-forge/noarch/font-ttf-source-code-pro-2.038-h77eed37_0.tar.bz2#4d59c254e01d9cde7957100457e2d5fb
+https://conda.anaconda.org/conda-forge/noarch/font-ttf-ubuntu-0.83-h77eed37_2.conda#cbbe59391138ea5ad3658c76912e147f
+https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.40-hf3520f5_2.conda#61b0bd5219ce7192b4e3633521a78975
+https://conda.anaconda.org/nvidia/linux-64/libcublas-12.1.0.26-0.tar.bz2#74f872929a02e01ef746a064fa46a80c
+https://conda.anaconda.org/nvidia/linux-64/libcufft-11.0.2.4-0.tar.bz2#b53f7ea28a363eb6d218bcbffb9d26aa
+https://conda.anaconda.org/nvidia/linux-64/libcusolver-11.4.4.55-0.tar.bz2#2d2fe4a7af91ec8a1eee7f1f0cf7b050
+https://conda.anaconda.org/nvidia/linux-64/libcusparse-12.0.2.55-0.tar.bz2#c295ea64ea0654af0cbe833431de6daa
+https://conda.anaconda.org/nvidia/linux-64/libnpp-12.0.2.50-0.tar.bz2#072e390c1e0e4909bdd7508dd6af1474
+https://conda.anaconda.org/nvidia/linux-64/libnvjpeg-12.1.1.14-0.tar.bz2#4dea93d43adfd03388b31f2ae9892558
+https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-ng-13.2.0-hc0a3c3a_7.conda#53ebd4c833fa01cb2c6353e99f905406
+https://conda.anaconda.org/conda-forge/linux-64/mkl-include-2022.1.0-h84fe81f_915.tar.bz2#2dcd1acca05c11410d4494d7fc7dfa2a
+https://conda.anaconda.org/conda-forge/linux-64/python_abi-3.12-4_cp312.conda#dccc2d142812964fcc6abdc97b672dff
+https://conda.anaconda.org/pytorch/noarch/pytorch-mutex-1.0-cuda.tar.bz2#a948316e36fb5b11223b3fcfa93f8358
+https://conda.anaconda.org/conda-forge/noarch/tzdata-2024a-h0c530f3_0.conda#161081fc7cec0bfda0d86d7cb595f8d8
+https://conda.anaconda.org/conda-forge/noarch/cuda-cudart_linux-64-12.1.105-h59595ed_0.conda#f8229a887df2311217d1528cc205073b
+https://conda.anaconda.org/conda-forge/noarch/fonts-conda-forge-1-0.tar.bz2#f766549260d6815b0c52253f1fb1bb29
+https://conda.anaconda.org/conda-forge/noarch/fonts-conda-ecosystem-1-0.tar.bz2#fee5683a3f04bd15cbd8318b096a27ab
+https://conda.anaconda.org/conda-forge/linux-64/_openmp_mutex-4.5-2_kmp_llvm.tar.bz2#562b26ba2e19059551a811e72ab7f793
+https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-13.2.0-h77fa898_7.conda#72ec1b1b04c4d15d4204ece1ecea5978
+https://conda.anaconda.org/conda-forge/linux-64/alsa-lib-1.2.11-hd590300_1.conda#0bb492cca54017ea314b809b1ee3a176
+https://conda.anaconda.org/conda-forge/linux-64/attr-2.5.1-h166bdaf_1.tar.bz2#d9c69a24ad678ffce24c6543a0176b00
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-common-0.9.19-h4ab18f5_0.conda#c6dedd5eab2236f4abb59ade9fb7fd44
+https://conda.anaconda.org/conda-forge/linux-64/bzip2-1.0.8-hd590300_5.conda#69b8b6202a07720f448be700e300ccf4
+https://conda.anaconda.org/conda-forge/linux-64/c-ares-1.28.1-hd590300_0.conda#dcde58ff9a1f30b0037a2315d1846d1f
+https://conda.anaconda.org/conda-forge/linux-64/cuda-cudart-12.1.105-hd3aeb46_0.conda#e2ab3aeff4d18c82b3e7025a2ec3cecc
+https://conda.anaconda.org/conda-forge/linux-64/cuda-cupti-12.1.105-h59595ed_0.conda#37400196a2a9d83a1a79ed763189ce32
+https://conda.anaconda.org/conda-forge/linux-64/cuda-nvrtc-12.1.105-hd3aeb46_0.conda#361041b17b31f25e60ac43127f52bd3a
+https://conda.anaconda.org/conda-forge/linux-64/cuda-nvtx-12.1.105-h59595ed_0.conda#a8e1192335156d6e0a8972794cd1da49
+https://conda.anaconda.org/conda-forge/linux-64/gettext-tools-0.22.5-h59595ed_2.conda#985f2f453fb72408d6b6f1be0f324033
+https://conda.anaconda.org/conda-forge/linux-64/gflags-2.2.2-he1b5a44_1004.tar.bz2#cddaf2c63ea4a5901cf09524c490ecdc
+https://conda.anaconda.org/conda-forge/linux-64/gmp-6.3.0-h59595ed_1.conda#e358c7c5f6824c272b5034b3816438a7
+https://conda.anaconda.org/conda-forge/linux-64/graphite2-1.3.13-h59595ed_1003.conda#f87c7b7c2cb45f323ffbce941c78ab7c
+https://conda.anaconda.org/conda-forge/linux-64/icu-73.2-h59595ed_0.conda#cc47e1facc155f91abd89b11e48e72ff
+https://conda.anaconda.org/conda-forge/linux-64/keyutils-1.6.1-h166bdaf_0.tar.bz2#30186d27e2c9fa62b45fb1476b7200e3
+https://conda.anaconda.org/conda-forge/linux-64/lame-3.100-h166bdaf_1003.tar.bz2#a8832b479f93521a9e7b5b743803be51
+https://conda.anaconda.org/conda-forge/linux-64/lerc-4.0.0-h27087fc_0.tar.bz2#76bbff344f0134279f225174e9064c8f
+https://conda.anaconda.org/conda-forge/linux-64/libabseil-20240116.2-cxx17_h59595ed_0.conda#682bdbe046a68f749769b492f3625c5c
+https://conda.anaconda.org/conda-forge/linux-64/libasprintf-0.22.5-h661eb56_2.conda#dd197c968bf9760bba0031888d431ede
+https://conda.anaconda.org/conda-forge/linux-64/libbrotlicommon-1.1.0-hd590300_1.conda#aec6c91c7371c26392a06708a73c70e5
+https://conda.anaconda.org/conda-forge/linux-64/libcrc32c-1.1.2-h9c3ff4c_0.tar.bz2#c965a5aa0d5c1c37ffc62dff36e28400
+https://conda.anaconda.org/conda-forge/linux-64/libcufile-1.6.1.9-hd3aeb46_0.conda#9a58d214028c01750eaa2cd07386150d
+https://conda.anaconda.org/conda-forge/linux-64/libcurand-10.3.2.106-hd3aeb46_0.conda#1bd892b578e3bfb7fb482c943ed3d904
+https://conda.anaconda.org/conda-forge/linux-64/libdeflate-1.20-hd590300_0.conda#8e88f9389f1165d7c0936fe40d9a9a79
+https://conda.anaconda.org/conda-forge/linux-64/libev-4.33-hd590300_2.conda#172bf1cd1ff8629f2b1179945ed45055
+https://conda.anaconda.org/conda-forge/linux-64/libexpat-2.6.2-h59595ed_0.conda#e7ba12deb7020dd080c6c70e7b6f6a3d
+https://conda.anaconda.org/conda-forge/linux-64/libffi-3.4.2-h7f98852_5.tar.bz2#d645c6d2ac96843a2bfaccd2d62b3ac3
+https://conda.anaconda.org/conda-forge/linux-64/libgettextpo-0.22.5-h59595ed_2.conda#172bcc51059416e7ce99e7b528cede83
+https://conda.anaconda.org/conda-forge/linux-64/libgfortran5-13.2.0-hca663fb_7.conda#c0bd771f09a326fdcd95a60b617795bf
+https://conda.anaconda.org/conda-forge/linux-64/libiconv-1.17-hd590300_2.conda#d66573916ffcf376178462f1b61c941e
+https://conda.anaconda.org/conda-forge/linux-64/libjpeg-turbo-3.0.0-hd590300_1.conda#ea25936bb4080d843790b586850f82b8
+https://conda.anaconda.org/conda-forge/linux-64/libnsl-2.0.1-hd590300_0.conda#30fd6e37fe21f86f4bd26d6ee73eeec7
+https://conda.anaconda.org/conda-forge/linux-64/libnvjitlink-12.1.105-hd3aeb46_0.conda#ed70b41cca6446cab43b0069bf17bd9c
+https://conda.anaconda.org/conda-forge/linux-64/libogg-1.3.4-h7f98852_1.tar.bz2#6e8cc2173440d77708196c5b93771680
+https://conda.anaconda.org/conda-forge/linux-64/libopus-1.3.1-h7f98852_1.tar.bz2#15345e56d527b330e1cacbdf58676e8f
+https://conda.anaconda.org/conda-forge/linux-64/libutf8proc-2.8.0-h166bdaf_0.tar.bz2#ede4266dc02e875fe1ea77b25dd43747
+https://conda.anaconda.org/conda-forge/linux-64/libuuid-2.38.1-h0b41bf4_0.conda#40b61aab5c7ba9ff276c41cfffe6b80b
+https://conda.anaconda.org/conda-forge/linux-64/libwebp-base-1.4.0-hd590300_0.conda#b26e8aa824079e1be0294e7152ca4559
+https://conda.anaconda.org/conda-forge/linux-64/libxcrypt-4.4.36-hd590300_1.conda#5aa797f8787fe7a17d1b0821485b5adc
+https://conda.anaconda.org/conda-forge/linux-64/libzlib-1.3.1-h4ab18f5_1.conda#57d7dc60e9325e3de37ff8dffd18e814
+https://conda.anaconda.org/conda-forge/linux-64/lz4-c-1.9.4-hcb278e6_0.conda#318b08df404f9c9be5712aaa5a6f0bb0
+https://conda.anaconda.org/conda-forge/linux-64/mpg123-1.32.6-h59595ed_0.conda#9160cdeb523a1b20cf8d2a0bf821f45d
+https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.5-h59595ed_0.conda#fcea371545eda051b6deafb24889fc69
+https://conda.anaconda.org/conda-forge/linux-64/ninja-1.12.1-h297d8ca_0.conda#3aa1c7e292afeff25a0091ddd7c69b72
+https://conda.anaconda.org/conda-forge/linux-64/nspr-4.35-h27087fc_0.conda#da0ec11a6454ae19bff5b02ed881a2b1
+https://conda.anaconda.org/conda-forge/linux-64/ocl-icd-2.3.2-hd590300_1.conda#c66f837ac65e4d1cdeb80e2a1d5fcc3d
+https://conda.anaconda.org/conda-forge/linux-64/openssl-3.3.0-h4ab18f5_3.conda#12ea6d0d4ed54530eaed18e4835c1f7c
+https://conda.anaconda.org/conda-forge/linux-64/pixman-0.43.2-h59595ed_0.conda#71004cbf7924e19c02746ccde9fd7123
+https://conda.anaconda.org/conda-forge/linux-64/pthread-stubs-0.4-h36c2ea0_1001.tar.bz2#22dad4df6e8630e8dff2428f6f6a7036
+https://conda.anaconda.org/conda-forge/linux-64/snappy-1.2.0-hdb0a2a9_1.conda#843bbb8ace1d64ac50d64639ff38b014
+https://conda.anaconda.org/conda-forge/linux-64/xorg-kbproto-1.0.7-h7f98852_1002.tar.bz2#4b230e8381279d76131116660f5a241a
+https://conda.anaconda.org/conda-forge/linux-64/xorg-libice-1.1.1-hd590300_0.conda#b462a33c0be1421532f28bfe8f4a7514
+https://conda.anaconda.org/conda-forge/linux-64/xorg-libxau-1.0.11-hd590300_0.conda#2c80dc38fface310c9bd81b17037fee5
+https://conda.anaconda.org/conda-forge/linux-64/xorg-libxdmcp-1.1.3-h7f98852_0.tar.bz2#be93aabceefa2fac576e971aef407908
+https://conda.anaconda.org/conda-forge/linux-64/xorg-renderproto-0.11.1-h7f98852_1002.tar.bz2#06feff3d2634e3097ce2fe681474b534
+https://conda.anaconda.org/conda-forge/linux-64/xorg-xextproto-7.3.0-h0b41bf4_1003.conda#bce9f945da8ad2ae9b1d7165a64d0f87
+https://conda.anaconda.org/conda-forge/linux-64/xorg-xf86vidmodeproto-2.3.1-h7f98852_1002.tar.bz2#3ceea9668625c18f19530de98b15d5b0
+https://conda.anaconda.org/conda-forge/linux-64/xorg-xproto-7.0.31-h7f98852_1007.tar.bz2#b4a4381d54784606820704f7b5f05a15
+https://conda.anaconda.org/conda-forge/linux-64/xz-5.2.6-h166bdaf_0.tar.bz2#2161070d867d1b1204ea749c8eec4ef0
+https://conda.anaconda.org/conda-forge/linux-64/yaml-0.2.5-h7f98852_2.tar.bz2#4cb3ad778ec2d5a7acbdf254eb1c42ae
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-cal-0.6.14-h88a6e22_1.conda#7ed63b0e816dd1635903506ef5d2c079
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-compression-0.2.18-h83b837d_6.conda#3e572eacd0ce99a59e1bb9c260ad5b20
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-sdkutils-0.1.16-h83b837d_2.conda#f40c698b4ea90f7fedd187c6639c818b
+https://conda.anaconda.org/conda-forge/linux-64/aws-checksums-0.1.18-h83b837d_6.conda#7995cb937bdac5913c8904fed6b3729d
+https://conda.anaconda.org/conda-forge/linux-64/cuda-opencl-12.1.105-h59595ed_0.conda#f2589b459bbf72de590aea9383a2568a
+https://conda.anaconda.org/conda-forge/linux-64/expat-2.6.2-h59595ed_0.conda#53fb86322bdb89496d7579fe3f02fd61
+https://conda.anaconda.org/conda-forge/linux-64/glog-0.7.0-hed5481d_0.conda#a9ea19c48e11754899299f8123070f4e
+https://conda.anaconda.org/conda-forge/linux-64/libasprintf-devel-0.22.5-h661eb56_2.conda#02e41ab5834dcdcc8590cf29d9526f50
+https://conda.anaconda.org/conda-forge/linux-64/libbrotlidec-1.1.0-hd590300_1.conda#f07002e225d7a60a694d42a7bf5ff53f
+https://conda.anaconda.org/conda-forge/linux-64/libbrotlienc-1.1.0-hd590300_1.conda#5fc11c6020d421960607d821310fcd4d
+https://conda.anaconda.org/conda-forge/linux-64/libcap-2.69-h0f662aa_0.conda#25cb5999faa414e5ccb2c1388f62d3d5
+https://conda.anaconda.org/conda-forge/linux-64/libedit-3.1.20191231-he28a2e2_2.tar.bz2#4d331e44109e3f0e19b4cb8f9b82f3e1
+https://conda.anaconda.org/conda-forge/linux-64/libevent-2.1.12-hf998b51_1.conda#a1cfcc585f0c42bf8d5546bb1dfb668d
+https://conda.anaconda.org/conda-forge/linux-64/libgettextpo-devel-0.22.5-h59595ed_2.conda#b63d9b6da3653179a278077f0de20014
+https://conda.anaconda.org/conda-forge/linux-64/libgfortran-ng-13.2.0-h69a702a_7.conda#1b84f26d9f4f6026e179e7805d5a15cd
+https://conda.anaconda.org/conda-forge/linux-64/libnghttp2-1.58.0-h47da74e_1.conda#700ac6ea6d53d5510591c4344d5c989a
+https://conda.anaconda.org/conda-forge/linux-64/libpng-1.6.43-h2797004_0.conda#009981dd9cfcaa4dbfa25ffaed86bcae
+https://conda.anaconda.org/conda-forge/linux-64/libprotobuf-4.25.3-h08a7969_0.conda#6945825cebd2aeb16af4c69d97c32c13
+https://conda.anaconda.org/conda-forge/linux-64/libre2-11-2023.09.01-h5a48ba9_2.conda#41c69fba59d495e8cf5ffda48a607e35
+https://conda.anaconda.org/conda-forge/linux-64/libsqlite-3.45.3-h2797004_0.conda#b3316cbe90249da4f8e84cd66e1cc55b
+https://conda.anaconda.org/conda-forge/linux-64/libssh2-1.11.0-h0841786_0.conda#1f5a58e686b13bcfde88b93f547d23fe
+https://conda.anaconda.org/conda-forge/linux-64/libvorbis-1.3.7-h9c3ff4c_0.tar.bz2#309dec04b70a3cc0f1e84a4013683bc0
+https://conda.anaconda.org/conda-forge/linux-64/libxcb-1.15-h0b41bf4_0.conda#33277193f5b92bad9fdd230eb700929c
+https://conda.anaconda.org/conda-forge/linux-64/libxml2-2.12.7-hc051c1a_0.conda#5d801a4906adc712d480afc362623b59
+https://conda.anaconda.org/conda-forge/linux-64/llvm-openmp-15.0.7-h0cdce71_0.conda#589c9a3575a050b583241c3d688ad9aa
+https://conda.anaconda.org/conda-forge/linux-64/mpfr-4.2.1-h9458935_1.conda#8083b20f566639c22f78bcd6ca35b276
+https://conda.anaconda.org/conda-forge/linux-64/mysql-common-8.3.0-hf1915f5_4.conda#784a4df6676c581ca624fbe460703a6d
+https://conda.anaconda.org/conda-forge/linux-64/pcre2-10.43-hcad00b1_0.conda#8292dea9e022d9610a11fce5e0896ed8
+https://conda.anaconda.org/conda-forge/linux-64/readline-8.2-h8228510_1.conda#47d31b792659ce70f470b5c82fdfb7a4
+https://conda.anaconda.org/conda-forge/linux-64/s2n-1.4.15-he19d79f_0.conda#4c7cc3fa1d2c5a63f9e2b1e2980a1672
+https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.13-noxft_h4845f30_101.conda#d453b98d9c83e71da0741bb0ff4d76bc
+https://conda.anaconda.org/conda-forge/linux-64/xorg-libsm-1.2.4-h7391055_0.conda#93ee23f12bc2e684548181256edd2cf6
+https://conda.anaconda.org/conda-forge/linux-64/zlib-1.3.1-h4ab18f5_1.conda#9653f1bf3766164d0e65fa723cabbc54
+https://conda.anaconda.org/conda-forge/linux-64/zstd-1.5.6-ha6fb4c9_0.conda#4d056880988120e29d75bfff282e0f45
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-io-0.14.8-h21d4f22_5.conda#f9dd6e8a46f55f49eae5380d3b922b71
+https://conda.anaconda.org/conda-forge/linux-64/brotli-bin-1.1.0-hd590300_1.conda#39f910d205726805a958da408ca194ba
+https://conda.anaconda.org/nvidia/linux-64/cuda-libraries-12.1.0-0.tar.bz2#8c08238819848e471a6213db526dbf15
+https://conda.anaconda.org/conda-forge/linux-64/freetype-2.12.1-h267a509_2.conda#9ae35c3d96db2c94ce0cef86efdfa2cb
+https://conda.anaconda.org/conda-forge/linux-64/gettext-0.22.5-h59595ed_2.conda#219ba82e95d7614cf7140d2a4afc0926
+https://conda.anaconda.org/conda-forge/linux-64/krb5-1.21.2-h659d440_0.conda#cd95826dbd331ed1be26bdf401432844
+https://conda.anaconda.org/conda-forge/linux-64/libglib-2.80.2-hf974151_0.conda#72724f6a78ecb15559396966226d5838
+https://conda.anaconda.org/conda-forge/linux-64/libhiredis-1.0.2-h2cc385e_0.tar.bz2#b34907d3a81a3cd8095ee83d174c074a
+https://conda.anaconda.org/conda-forge/linux-64/libhwloc-2.10.0-default_h5622ce7_1001.conda#fc2d5b79c2d3f8568fbab31db7ae02f3
+https://conda.anaconda.org/conda-forge/linux-64/libllvm15-15.0.7-hb3ce162_4.conda#8a35df3cbc0c8b12cc8af9473ae75eef
+https://conda.anaconda.org/conda-forge/linux-64/libllvm18-18.1.6-hb77312f_0.conda#1246fc4b9f4db452e69cc297967d4b3e
+https://conda.anaconda.org/conda-forge/linux-64/libthrift-0.19.0-hb90f79a_1.conda#8cdb7d41faa0260875ba92414c487e2d
+https://conda.anaconda.org/conda-forge/linux-64/libtiff-4.6.0-h1dd3fc0_3.conda#66f03896ffbe1a110ffda05c7a856504
+https://conda.anaconda.org/conda-forge/linux-64/mpc-1.3.1-hfe3b2da_0.conda#289c71e83dc0daa7d4c81f04180778ca
+https://conda.anaconda.org/conda-forge/linux-64/mysql-libs-8.3.0-hca2cd23_4.conda#1b50eebe2a738a3146c154d2eceaa8b6
+https://conda.anaconda.org/conda-forge/linux-64/nss-3.100-hca3bf56_0.conda#949c4a82290ee58b3c970cef4bcfd4ad
+https://conda.anaconda.org/conda-forge/linux-64/orc-2.0.1-h17fec99_1.conda#3bf65f0d8e7322a1cfe8b670fa35ec81
+https://conda.anaconda.org/conda-forge/linux-64/python-3.12.3-hab00c5b_0_cpython.conda#2540b74d304f71d3e89c81209db4db84
+https://conda.anaconda.org/conda-forge/linux-64/re2-2023.09.01-h7f4b329_2.conda#8f70e36268dea8eb666ef14c29bd3cda
+https://conda.anaconda.org/conda-forge/linux-64/xcb-util-0.4.0-hd590300_1.conda#9bfac7ccd94d54fd21a0501296d60424
+https://conda.anaconda.org/conda-forge/linux-64/xcb-util-keysyms-0.4.0-h8ee46fc_1.conda#632413adcd8bc16b515cab87a2932913
+https://conda.anaconda.org/conda-forge/linux-64/xcb-util-renderutil-0.3.9-hd590300_1.conda#e995b155d938b6779da6ace6c6b13816
+https://conda.anaconda.org/conda-forge/linux-64/xcb-util-wm-0.4.1-h8ee46fc_1.conda#90108a432fb5c6150ccfee3f03388656
+https://conda.anaconda.org/conda-forge/linux-64/xorg-libx11-1.8.9-h8ee46fc_0.conda#077b6e8ad6a3ddb741fce2496dd01bec
+https://conda.anaconda.org/conda-forge/noarch/array-api-compat-1.7.1-pyhd8ed1ab_0.conda#8791d81c38f676a7c08c76546800bf70
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-event-stream-0.4.2-ha47c788_12.conda#8420d8e495a1468f593128e5fbf6748a
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-http-0.8.1-h29d6fba_17.conda#c20a29ff47043ba1ec24f45dc68930bf
+https://conda.anaconda.org/conda-forge/linux-64/brotli-1.1.0-hd590300_1.conda#f27a24d46e3ea7b70a1f98e50c62508f
+https://conda.anaconda.org/conda-forge/linux-64/ccache-4.9.1-h1fcd64f_0.conda#3620f564bcf28c3524951b6f64f5c5ac
+https://conda.anaconda.org/conda-forge/noarch/certifi-2024.2.2-pyhd8ed1ab_0.conda#0876280e409658fc6f9e75d035960333
+https://conda.anaconda.org/conda-forge/noarch/colorama-0.4.6-pyhd8ed1ab_0.tar.bz2#3faab06a954c2a04039983f2c4a50d99
+https://conda.anaconda.org/nvidia/linux-64/cuda-runtime-12.1.0-0.tar.bz2#95e8c2f09ec28cce7cdecd6200b5d26e
+https://conda.anaconda.org/conda-forge/noarch/cycler-0.12.1-pyhd8ed1ab_0.conda#5cd86562580f274031ede6aa6aa24441
+https://conda.anaconda.org/conda-forge/linux-64/cython-3.0.10-py312h30efb56_0.conda#b119273bff37284cbcb9281c1e85e67d
+https://conda.anaconda.org/conda-forge/linux-64/dbus-1.13.6-h5008d03_3.tar.bz2#ecfff944ba3960ecb334b9a2663d708d
+https://conda.anaconda.org/conda-forge/noarch/exceptiongroup-1.2.0-pyhd8ed1ab_2.conda#8d652ea2ee8eaee02ed8dc820bc794aa
+https://conda.anaconda.org/conda-forge/noarch/execnet-2.1.1-pyhd8ed1ab_0.conda#15dda3cdbf330abfe9f555d22f66db46
+https://conda.anaconda.org/conda-forge/linux-64/fastrlock-0.8.2-py312h30efb56_2.conda#7065ec5a4909f925e305b77e505b0aec
+https://conda.anaconda.org/conda-forge/noarch/filelock-3.14.0-pyhd8ed1ab_0.conda#831d85ae0acfba31b8efd0f0d07da736
+https://conda.anaconda.org/conda-forge/linux-64/fontconfig-2.14.2-h14ed4e7_0.conda#0f69b688f52ff6da70bccb7ff7001d1d
+https://conda.anaconda.org/conda-forge/linux-64/glib-tools-2.80.2-hb6ce0ca_0.conda#a965aeaf060289528a3fbe09326edae2
+https://conda.anaconda.org/conda-forge/linux-64/gmpy2-2.1.5-py312h1d5cde6_1.conda#27abd7664bc87595bd98b6306b8393d1
+https://conda.anaconda.org/conda-forge/noarch/iniconfig-2.0.0-pyhd8ed1ab_0.conda#f800d2da156d08e289b14e87e43c1ae5
+https://conda.anaconda.org/conda-forge/linux-64/kiwisolver-1.4.5-py312h8572e83_1.conda#c1e71f2bc05d8e8e033aefac2c490d05
+https://conda.anaconda.org/conda-forge/linux-64/lcms2-2.16-hb7c19ff_0.conda#51bb7010fc86f70eee639b4bb7a894f5
+https://conda.anaconda.org/conda-forge/linux-64/libclang-cpp15-15.0.7-default_h127d8a8_5.conda#d0a9633b53cdc319b8a1a532ae7822b8
+https://conda.anaconda.org/conda-forge/linux-64/libclang13-18.1.6-default_h5d6823c_0.conda#fbe666f653068958eb27f549cb12f202
+https://conda.anaconda.org/conda-forge/linux-64/libcups-2.3.3-h4637d8d_4.conda#d4529f4dff3057982a7617c7ac58fde3
+https://conda.anaconda.org/conda-forge/linux-64/libcurl-8.8.0-hca28451_0.conda#f21c27f076a07907e70c49bb57bd0f20
+https://conda.anaconda.org/conda-forge/linux-64/libflac-1.4.3-h59595ed_0.conda#ee48bf17cc83a00f59ca1494d5646869
+https://conda.anaconda.org/conda-forge/linux-64/libgpg-error-1.49-h4f305b6_0.conda#dfcfd72c7a430d3616763ecfbefe4ca9
+https://conda.anaconda.org/conda-forge/linux-64/libgrpc-1.62.2-h15f2491_0.conda#8dabe607748cb3d7002ad73cd06f1325
+https://conda.anaconda.org/conda-forge/linux-64/libpq-16.3-ha72fbe1_0.conda#bac737ae28b79cfbafd515258d97d29e
+https://conda.anaconda.org/conda-forge/linux-64/markupsafe-2.1.5-py312h98912ed_0.conda#6ff0b9582da2d4a74a1f9ae1f9ce2af6
+https://conda.anaconda.org/conda-forge/noarch/mpmath-1.3.0-pyhd8ed1ab_0.conda#dbf6e2d89137da32fa6670f3bffc024e
+https://conda.anaconda.org/conda-forge/noarch/munkres-1.1.4-pyh9f0ad1d_0.tar.bz2#2ba8498c1018c1e9c61eb99b973dfe19
+https://conda.anaconda.org/conda-forge/noarch/networkx-3.3-pyhd8ed1ab_1.conda#d335fd5704b46f4efb89a6774e81aef0
+https://conda.anaconda.org/conda-forge/linux-64/openjpeg-2.5.2-h488ebb8_0.conda#7f2e286780f072ed750df46dc2631138
+https://conda.anaconda.org/conda-forge/noarch/packaging-24.0-pyhd8ed1ab_0.conda#248f521b64ce055e7feae3105e7abeb8
+https://conda.anaconda.org/conda-forge/noarch/pluggy-1.5.0-pyhd8ed1ab_0.conda#d3483c8fc2dc2cc3f5cf43e26d60cabf
+https://conda.anaconda.org/conda-forge/noarch/ply-3.11-pyhd8ed1ab_2.conda#18c6deb6f9602e32446398203c8f0e91
+https://conda.anaconda.org/conda-forge/noarch/pyparsing-3.1.2-pyhd8ed1ab_0.conda#b9a4dacf97241704529131a0dfc0494f
+https://conda.anaconda.org/conda-forge/noarch/python-tzdata-2024.1-pyhd8ed1ab_0.conda#98206ea9954216ee7540f0c773f2104d
+https://conda.anaconda.org/conda-forge/noarch/pytz-2024.1-pyhd8ed1ab_0.conda#3eeeeb9e4827ace8c0c1419c85d590ad
+https://conda.anaconda.org/conda-forge/linux-64/pyyaml-6.0.1-py312h98912ed_1.conda#e3fd78d8d490af1d84763b9fe3f2e552
+https://conda.anaconda.org/conda-forge/noarch/setuptools-70.0.0-pyhd8ed1ab_0.conda#c8ddb4f34a208df4dd42509a0f6a1c89
+https://conda.anaconda.org/conda-forge/noarch/six-1.16.0-pyh6c4a22f_0.tar.bz2#e5f25f8dbc060e9a8d912e432202afc2
+https://conda.anaconda.org/conda-forge/linux-64/tbb-2021.12.0-h297d8ca_1.conda#3ff978d8994f591818a506640c6a7071
+https://conda.anaconda.org/conda-forge/noarch/threadpoolctl-3.5.0-pyhc1e730c_0.conda#df68d78237980a159bd7149f33c0e8fd
+https://conda.anaconda.org/conda-forge/noarch/toml-0.10.2-pyhd8ed1ab_0.tar.bz2#f832c45a477c78bebd107098db465095
+https://conda.anaconda.org/conda-forge/noarch/tomli-2.0.1-pyhd8ed1ab_0.tar.bz2#5844808ffab9ebdb694585b50ba02a96
+https://conda.anaconda.org/conda-forge/linux-64/tornado-6.4-py312h98912ed_0.conda#e8332e534dca8c5c12c8352e0a23501c
+https://conda.anaconda.org/conda-forge/noarch/typing_extensions-4.12.1-pyha770c72_0.conda#26d7ee34132362115093717c706c384c
+https://conda.anaconda.org/conda-forge/noarch/wheel-0.43.0-pyhd8ed1ab_1.conda#0b5293a157c2b5cd513dd1b03d8d3aae
+https://conda.anaconda.org/conda-forge/linux-64/xcb-util-image-0.4.0-h8ee46fc_1.conda#9d7bcddf49cbf727730af10e71022c73
+https://conda.anaconda.org/conda-forge/linux-64/xkeyboard-config-2.41-hd590300_0.conda#81f740407b45e3f9047b3174fa94eb9e
+https://conda.anaconda.org/conda-forge/linux-64/xorg-libxext-1.3.4-h0b41bf4_2.conda#82b6df12252e6f32402b96dacc656fec
+https://conda.anaconda.org/conda-forge/linux-64/xorg-libxrender-0.9.11-hd590300_0.conda#ed67c36f215b310412b2af935bf3e530
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-auth-0.7.22-h96bc93b_2.conda#de2b7c9aa9b279cca5542134b7a2b86a
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-mqtt-0.10.4-h759edc4_4.conda#8ced661d9dcece8698922fd8a73b6511
+https://conda.anaconda.org/conda-forge/linux-64/cairo-1.18.0-h3faef2a_0.conda#f907bb958910dc404647326ca80c263e
+https://conda.anaconda.org/conda-forge/linux-64/coverage-7.5.3-py312h9a8786e_0.conda#f01930d0afe8ac5f8062c98e6b8d1fd0
+https://conda.anaconda.org/conda-forge/linux-64/fonttools-4.53.0-py312h9a8786e_0.conda#8490346e9d5efd7a6869582aa0c95b25
+https://conda.anaconda.org/conda-forge/linux-64/glib-2.80.2-hf974151_0.conda#d427988dc3dbd0a4c136f52db356cc6a
+https://conda.anaconda.org/conda-forge/noarch/jinja2-3.1.4-pyhd8ed1ab_0.conda#7b86ecb7d3557821c649b3c31e3eb9f2
+https://conda.anaconda.org/conda-forge/noarch/joblib-1.4.2-pyhd8ed1ab_0.conda#25df261d4523d9f9783bcdb7208d872f
+https://conda.anaconda.org/conda-forge/linux-64/libgcrypt-1.10.3-hd590300_0.conda#32d16ad533c59bb0a3c5ffaf16110829
+https://conda.anaconda.org/conda-forge/linux-64/libgoogle-cloud-2.24.0-h2736e30_0.conda#34aeee3fa7fca5dc21fad3ac6f4f0ab2
+https://conda.anaconda.org/conda-forge/linux-64/libsndfile-1.2.2-hc60ed4a_1.conda#ef1910918dd895516a769ed36b5b3a4e
+https://conda.anaconda.org/conda-forge/linux-64/libxkbcommon-1.7.0-h662e7e4_0.conda#b32c0da42b1f24a98577bb3d7fc0b995
+https://conda.anaconda.org/conda-forge/noarch/meson-1.4.0-pyhd8ed1ab_0.conda#52a0660cfa40b45bf254ecc3374cb2e0
+https://conda.anaconda.org/conda-forge/linux-64/mkl-2022.1.0-h84fe81f_915.tar.bz2#b9c8f925797a93dbff45e1626b025a6b
+https://conda.anaconda.org/conda-forge/linux-64/pillow-10.3.0-py312hdcec9eb_0.conda#425bb325f970e57a047ac57c4586489d
+https://conda.anaconda.org/conda-forge/noarch/pip-24.0-pyhd8ed1ab_0.conda#f586ac1e56c8638b64f9c8122a7b8a67
+https://conda.anaconda.org/conda-forge/noarch/pyproject-metadata-0.8.0-pyhd8ed1ab_0.conda#573fe09d7bd0cd4bcc210d8369b5ca47
+https://conda.anaconda.org/conda-forge/noarch/pytest-8.2.1-pyhd8ed1ab_0.conda#e4418e8bdbaa8eea28e047531e6763c8
+https://conda.anaconda.org/conda-forge/noarch/python-dateutil-2.9.0-pyhd8ed1ab_0.conda#2cf4264fffb9e6eff6031c5b6884d61c
+https://conda.anaconda.org/pytorch/linux-64/pytorch-cuda-12.1-ha16c6d3_5.tar.bz2#ffc0937cf6ba3ffb299b0c256accc53f
+https://conda.anaconda.org/conda-forge/linux-64/sip-6.7.12-py312h30efb56_0.conda#32633871002ee9902f747d2236e0d122
+https://conda.anaconda.org/conda-forge/noarch/sympy-1.12-pypyh9d50eac_103.conda#2f7d6347d7acf6edf1ac7f2189f44c8f
+https://conda.anaconda.org/conda-forge/linux-64/aws-c-s3-0.5.9-h594631b_3.conda#47490db1dcddfb1c355251fc427746a6
+https://conda.anaconda.org/conda-forge/linux-64/gstreamer-1.24.4-haf2f30d_0.conda#926c2c7ee7a0b48d6d70783a33f7bc80
+https://conda.anaconda.org/conda-forge/linux-64/harfbuzz-8.5.0-hfac3d4d_0.conda#f5126317dd0ce0ba26945e411ecc6960
+https://conda.anaconda.org/conda-forge/linux-64/libblas-3.9.0-16_linux64_mkl.tar.bz2#85f61af03fd291dae33150ffe89dc09a
+https://conda.anaconda.org/conda-forge/linux-64/libgoogle-cloud-storage-2.24.0-h3d9a0c8_0.conda#a731371833a7b1ab3a87be0fe7e6235a
+https://conda.anaconda.org/conda-forge/linux-64/libsystemd0-255-h3516f8a_1.conda#3366af27f0b593544a6cd453c7932ac5
+https://conda.anaconda.org/conda-forge/noarch/meson-python-0.16.0-pyh0c530f3_0.conda#e16f0dbf502da873be9f9adb0dc52547
+https://conda.anaconda.org/conda-forge/linux-64/mkl-devel-2022.1.0-ha770c72_916.tar.bz2#69ba49e445f87aea2cba343a71a35ca2
+https://conda.anaconda.org/conda-forge/linux-64/pyqt5-sip-12.12.2-py312h30efb56_5.conda#8a2a122dc4fe14d8cff38f1cf426381f
+https://conda.anaconda.org/conda-forge/noarch/pytest-cov-5.0.0-pyhd8ed1ab_0.conda#c54c0107057d67ddf077751339ec2c63
+https://conda.anaconda.org/conda-forge/noarch/pytest-xdist-3.5.0-pyhd8ed1ab_0.conda#d5f595da2daead898ca958ac62f0307b
+https://conda.anaconda.org/conda-forge/linux-64/aws-crt-cpp-0.26.9-he3a8b3b_0.conda#fbe6a256dd70a505730e7c461cd37a35
+https://conda.anaconda.org/conda-forge/linux-64/gst-plugins-base-1.24.4-h9ad1361_0.conda#147cce520ec59367549fd0d96d404213
+https://conda.anaconda.org/conda-forge/linux-64/libcblas-3.9.0-16_linux64_mkl.tar.bz2#361bf757b95488de76c4f123805742d3
+https://conda.anaconda.org/conda-forge/linux-64/liblapack-3.9.0-16_linux64_mkl.tar.bz2#a2f166748917d6d6e4707841ca1f519e
+https://conda.anaconda.org/conda-forge/linux-64/pulseaudio-client-17.0-hb77b528_0.conda#07f45f1be1c25345faddb8db0de8039b
+https://conda.anaconda.org/conda-forge/linux-64/aws-sdk-cpp-1.11.329-hba8bd5f_3.conda#720494d9f06b4aff1270cffb7acc7920
+https://conda.anaconda.org/conda-forge/linux-64/liblapacke-3.9.0-16_linux64_mkl.tar.bz2#44ccc4d4dca6a8d57fa17442bc64b5a1
+https://conda.anaconda.org/conda-forge/linux-64/numpy-1.26.4-py312heda63a1_0.conda#d8285bea2a350f63fab23bf460221f3f
+https://conda.anaconda.org/conda-forge/linux-64/qt-main-5.15.8-hc9dc06e_21.conda#b325046180590c868ce0dbf267b82eb8
+https://conda.anaconda.org/conda-forge/noarch/array-api-strict-1.1.1-pyhd8ed1ab_0.conda#941bbcd64d1a7b44aeb497f468fc85b4
+https://conda.anaconda.org/conda-forge/linux-64/blas-devel-3.9.0-16_linux64_mkl.tar.bz2#3f92c1c9e1c0e183462c5071aa02cae1
+https://conda.anaconda.org/conda-forge/linux-64/contourpy-1.2.1-py312h8572e83_0.conda#12c6a831ef734f0b2dd4caff514cbb7f
+https://conda.anaconda.org/conda-forge/linux-64/cupy-core-13.1.0-py312hffdfcc6_4.conda#37a04419e4446e5486e06b85df58f1e7
+https://conda.anaconda.org/conda-forge/linux-64/libarrow-16.1.0-hcb6531f_6_cpu.conda#0df3fc2a8d63b1cc49973c5a679ec438
+https://conda.anaconda.org/conda-forge/linux-64/pandas-2.2.2-py312h1d6d2e6_1.conda#ae00b61f3000d2284d1f2584d4dfafa8
+https://conda.anaconda.org/conda-forge/linux-64/polars-0.20.31-py312hc7f843c_0.conda#c37ecb115967f1056ec360708913fdf1
+https://conda.anaconda.org/conda-forge/linux-64/pyqt-5.15.9-py312h949fe66_5.conda#f6548a564e2d01b2a42020259503945b
+https://conda.anaconda.org/conda-forge/linux-64/scipy-1.13.1-py312hc2bc53b_0.conda#864b2399a9c998e17d1a9a4e0c601285
+https://conda.anaconda.org/conda-forge/linux-64/blas-2.116-mkl.tar.bz2#c196a26abf6b4f132c88828ab7c2231c
+https://conda.anaconda.org/conda-forge/linux-64/cupy-13.1.0-py312h7b0f9d9_4.conda#630f021ce783be0a40afc7013ec4c6ed
+https://conda.anaconda.org/conda-forge/linux-64/libarrow-acero-16.1.0-hac33072_6_cpu.conda#38b1161e2f8c72095f64ea35ee1294c5
+https://conda.anaconda.org/conda-forge/linux-64/libparquet-16.1.0-h6a7eafb_6_cpu.conda#87f676c6cb33f8e1956948ee216fa3a1
+https://conda.anaconda.org/conda-forge/linux-64/matplotlib-base-3.8.4-py312h20ab3a6_2.conda#fbfe798f83f0d66410903ad8f40d5283
+https://conda.anaconda.org/conda-forge/linux-64/pyamg-5.1.0-py312h389efb2_1.conda#323587ece55d7578e88b37fb43e91ac6
+https://conda.anaconda.org/conda-forge/linux-64/pyarrow-core-16.1.0-py312h5429d62_1_cpu.conda#cee0cddfaedfd3657f429318207e5816
+https://conda.anaconda.org/conda-forge/linux-64/libarrow-dataset-16.1.0-hac33072_6_cpu.conda#2e9430df8ffd645a5bc7edffb252c3de
+https://conda.anaconda.org/conda-forge/linux-64/matplotlib-3.8.4-py312h7900ff3_2.conda#ac26198045dff11c94202bb3e1bdc132
+https://conda.anaconda.org/pytorch/linux-64/pytorch-2.3.1-py3.12_cuda12.1_cudnn8.9.2_0.tar.bz2#8806dd010a45f7eb4af40a24ff99de47
+https://conda.anaconda.org/conda-forge/linux-64/libarrow-substrait-16.1.0-h7e0c224_6_cpu.conda#81fea801c4bb126509e784cbd2ca4d17
+https://conda.anaconda.org/conda-forge/linux-64/pyarrow-16.1.0-py312h8da182e_1.conda#2d8b51007ba9ec982067ecfc74315c3a
diff --git a/build_tools/github/pylatest_conda_forge_cuda_array-api_linux-64_environment.yml b/build_tools/github/pylatest_conda_forge_cuda_array-api_linux-64_environment.yml
new file mode 100644
index 0000000000000..e2ffb1429aa1d
--- /dev/null
+++ b/build_tools/github/pylatest_conda_forge_cuda_array-api_linux-64_environment.yml
@@ -0,0 +1,34 @@
+# DO NOT EDIT: this file is generated from the specification found in the
+# following script to centralize the configuration for CI builds:
+# build_tools/update_environments_and_lock_files.py
+channels:
+  - conda-forge
+  - pytorch
+  - nvidia
+dependencies:
+  - python
+  - numpy
+  - blas
+  - scipy
+  - cython
+  - joblib
+  - threadpoolctl
+  - matplotlib
+  - pandas
+  - pyamg
+  - pytest
+  - pytest-xdist
+  - pillow
+  - pip
+  - ninja
+  - meson-python
+  - pytest-cov
+  - coverage
+  - ccache
+  - pytorch::pytorch
+  - pytorch-cuda
+  - polars
+  - pyarrow
+  - cupy
+  - array-api-compat
+  - array-api-strict