Repository: rapidsai/cuml
Pull request #6377: CI: Run extended accel mode tests
Author: login: csadorf
Labels: name: tests
description: Unit testing for project
name: improvement
description: Improvement / enhancement to an existing function
name: non-breaking
description: Non-breaking change
name: ci
Created: 2025-02-27T22:00:19Z | Merged: 2025-03-13T18:35:13Z



Diff:
diff --git a/.github/workflows/pr.yaml b/.github/workflows/pr.yaml
index 26b181f86f..3397124298 100644
--- a/.github/workflows/pr.yaml
+++ b/.github/workflows/pr.yaml
@@ -23,6 +23,7 @@ jobs:
       - conda-python-build
       - conda-python-tests-singlegpu
       - conda-python-tests-dask
+      - conda-python-scikit-learn-accel-tests
       - conda-notebook-tests
       - docs-build
       - telemetry-setup
@@ -162,6 +163,14 @@ jobs:
     with:
       build_type: pull-request
       script: "ci/test_python_dask.sh"
+  conda-python-scikit-learn-accel-tests:
+    needs: [conda-python-build, changed-files]
+    secrets: inherit
+    uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-25.04
+    if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python
+    with:
+      build_type: pull-request
+      script: "ci/test_python_scikit_learn_tests.sh"
   conda-notebook-tests:
     needs: [conda-python-build, changed-files]
     secrets: inherit
diff --git a/.gitignore b/.gitignore
index e7f8328d0f..003894b878 100644
--- a/.gitignore
+++ b/.gitignore
@@ -33,6 +33,8 @@ tmp/
 wheels/
 wheelhouse/
 _skbuild/
+junit-*.xml
+
 
 ## files pickled in notebook when ran during python docstring generation
 docs/source/*.model
@@ -73,3 +75,6 @@ compile_commands.json
 # ref: https://github.com/rapidsai/cuml/pull/6201
 pytest.ini
 !python/cuml/cuml/benchmark/automated/pytest.ini
+
+# testing
+ci/**/scikit-learn/
diff --git a/ci/accel/scikit-learn-tests/README.md b/ci/accel/scikit-learn-tests/README.md
new file mode 100644
index 0000000000..ba51b8aa5e
--- /dev/null
+++ b/ci/accel/scikit-learn-tests/README.md
@@ -0,0 +1,40 @@
+# scikit-learn Acceleration Tests
+
+This suite provides infrastructure to run and analyze tests for scikit-learn with cuML acceleration support.
+
+## Components
+
+- `run-tests.sh`
+  Executes scikit-learn tests using GPU-accelerated paths. Any arguments passed to the script are forwarded directly to pytest.
+
+  Example usage:
+  ```bash
+  ./run-tests.sh                     # Run all tests
+  ./run-tests.sh -v -k test_kmeans   # Run specific test with verbosity
+  ./run-tests.sh -x --pdb            # Stop on first failure and debug
+  ```
+
+- `summarize-results.sh`
+  Analyzes test results from an XML report file and prints a summary.
+  Options:
+  - `-v, --verbose`          : Display detailed failure information
+  - `-f, --fail-below VALUE`  : Set a minimum pass rate threshold (0-100)
+
+## Usage
+
+### 1. Run tests
+Run all tests:
+```bash
+./run-tests.sh
+```
+
+Run specific tests using pytest arguments:
+```bash
+./run-tests.sh -v -k "test_logistic"
+```
+
+### 2. Summarize test results
+Generate a summary from the XML report with a pass rate threshold:
+```bash
+./summarize-results.sh -v -f 80 report.xml
+```
diff --git a/ci/accel/scikit-learn-tests/run-tests.sh b/ci/accel/scikit-learn-tests/run-tests.sh
new file mode 100755
index 0000000000..6d429781ca
--- /dev/null
+++ b/ci/accel/scikit-learn-tests/run-tests.sh
@@ -0,0 +1,14 @@
+#!/bin/bash
+# Copyright (c) 2025, NVIDIA CORPORATION.
+
+# This script runs scikit-learn tests with the cuml.accel plugin.
+# Any arguments passed to this script are forwarded directly to pytest.
+#
+# Example usage:
+#   ./run-tests.sh                     # Run all tests
+#   ./run-tests.sh -v -k test_kmeans   # Run specific test with verbosity
+#   ./run-tests.sh -x --pdb            # Stop on first failure and debug
+
+set -eu
+
+pytest -p cuml.accel --pyargs sklearn -v $@
diff --git a/ci/accel/scikit-learn-tests/summarize-results.sh b/ci/accel/scikit-learn-tests/summarize-results.sh
new file mode 100755
index 0000000000..b92c4dec48
--- /dev/null
+++ b/ci/accel/scikit-learn-tests/summarize-results.sh
@@ -0,0 +1,95 @@
+#!/bin/bash
+# Copyright (c) 2025, NVIDIA CORPORATION.
+
+usage() {
+    echo "Usage: $0 [options] REPORT_FILE"
+    echo ""
+    echo "Options:"
+    echo "  -h, --help              Show this help message"
+    echo "  -v, --verbose           Show detailed failure information"
+    echo "  -f, --fail-below VALUE  Minimum pass rate threshold [0-100] (default: 0)"
+    exit 1
+}
+
+# Parse command line arguments
+THRESHOLD=0
+VERBOSE=0
+
+while [[ $# -gt 0 ]]; do
+    case $1 in
+        -h|--help)
+            usage
+            ;;
+        -f|--fail-below)
+            THRESHOLD="$2"
+            shift 2
+            ;;
+        -v|--verbose)
+            VERBOSE=1
+            shift
+            ;;
+        *)
+            if [ -z "${REPORT_FILE:-}" ]; then
+                REPORT_FILE="$1"
+            else
+                echo "Unknown option: $1"
+                usage
+            fi
+            shift
+            ;;
+    esac
+done
+
+if [ -z "${REPORT_FILE:-}" ]; then
+    echo "Error: No report file specified"
+    usage
+fi
+
+# Validate threshold is a number between 0 and 100
+if ! [[ "$THRESHOLD" =~ ^[0-9]+(\.[0-9]+)?$ ]]; then
+    echo "Error: Threshold must be a number"
+    exit 1
+fi
+
+if ! awk -v t="$THRESHOLD" 'BEGIN{exit !(t >= 0 && t <= 100)}'; then
+    echo "Error: Threshold must be between 0 and 100"
+    exit 1
+fi
+
+# Extract test statistics using xmllint
+total_tests=$(xmllint --xpath "string(/testsuites/testsuite/@tests)" "${REPORT_FILE}")
+failures=$(xmllint --xpath "string(/testsuites/testsuite/@failures)" "${REPORT_FILE}")
+errors=$(xmllint --xpath "string(/testsuites/testsuite/@errors)" "${REPORT_FILE}")
+skipped=$(xmllint --xpath "string(/testsuites/testsuite/@skipped)" "${REPORT_FILE}")
+time=$(xmllint --xpath "string(/testsuites/testsuite/@time)" "${REPORT_FILE}")
+
+# Calculate passed tests and pass rate using awk
+passed=$((total_tests - failures - errors - skipped))
+pass_rate=$(awk -v passed="$passed" -v total="$total_tests" 'BEGIN { printf "%.2f", (passed/total) * 100 }')
+
+# Print summary
+echo "Test Summary:"
+echo "  Total Tests: ${total_tests}"
+echo "  Passed:      ${passed}"
+echo "  Failed:      ${failures}"
+echo "  Errors:      ${errors}"
+echo "  Skipped:     ${skipped}"
+echo "  Pass Rate:   ${pass_rate}%"
+echo "  Total Time:  ${time}s"
+
+# List failed tests only in verbose mode
+if [ "$((failures + errors))" -gt 0 ] && [ "${VERBOSE}" -eq 1 ]; then
+    echo ""
+    echo "Failed Tests:"
+    xmllint --xpath "//testcase[failure or error]/@name" "${REPORT_FILE}" | tr ' ' '\n' | sed 's/name=//g' | sed 's/"//g' | grep .
+fi
+
+# Check if threshold is nonzero before applying the check.
+if awk -v rate="$pass_rate" -v threshold="$THRESHOLD" 'BEGIN { exit (rate >= threshold) }'; then
+    echo ""
+    echo "Error: Pass rate ${pass_rate}% is below threshold ${THRESHOLD}%"
+    exit 1
+fi
+
+# In all other cases, return with success code.
+exit 0
diff --git a/ci/test_python_scikit_learn_tests.sh b/ci/test_python_scikit_learn_tests.sh
new file mode 100755
index 0000000000..8ba2669d96
--- /dev/null
+++ b/ci/test_python_scikit_learn_tests.sh
@@ -0,0 +1,28 @@
+#!/bin/bash
+# Copyright (c) 2025, NVIDIA CORPORATION.
+
+# Support invoking test script outside the script directory
+cd "$(dirname "$(realpath "${BASH_SOURCE[0]}")")"/../
+
+# Common setup steps shared by Python test jobs
+source ./ci/test_python_common.sh
+
+EXITCODE=0
+trap "EXITCODE=1" ERR
+set +e
+
+# Run scikit-learn tests with acceleration enabled
+rapids-logger "Running scikit-learn tests with cuML acceleration"
+
+# Run the tests
+./ci/accel/scikit-learn-tests/run-tests.sh \
+    --junitxml="${RAPIDS_TESTS_DIR}/junit-cuml-accel-scikit-learn.xml" || true
+
+# Analyze results and check pass rate threshold
+rapids-logger "Analyzing test results"
+./ci/accel/scikit-learn-tests/summarize-results.sh \
+    --fail-below 80 \
+    "${RAPIDS_TESTS_DIR}/junit-cuml-accel-scikit-learn.xml"
+
+rapids-logger "Test script exiting with value: $EXITCODE"
+exit ${EXITCODE}


Reviews:
user: login: jcrist
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-11T19:40:42Z
user: login: csadorf
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T20:09:12Z
user: login: csadorf
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T16:06:10Z
user: login: bdice
body: Flushing comments from earlier — forgot to submit. I will re-review soon.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T18:33:03Z
user: login: bdice
body: Looks good. The latest changes are much clearer. Apologies for the delayed review.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T18:02:43Z

Review comments:
diff_hunk: @@ -0,0 +1,59 @@
+# scikit-learn Acceleration Tests
+
+This suite provides infrastructure to build, run, and analyze tests for scikit-learn with cuML acceleration support.
+
+## Components
+
+- `build.sh`
+  Clones and builds the scikit-learn repository.
+  Options:
+  - `--scikit-learn-version` : Specify the scikit-learn version to test (default: 1.5.2)
path: ci/accel/scikit-learn-tests/README.md
user: login: bdice
body: I would avoid hardcoded versions in this README. It's easy for these things to slip out of date.
created_at: 2025-03-10T20:57:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,57 @@
+#!/bin/bash
+# Copyright (c) 2025, NVIDIA CORPORATION.
+
+usage() {
+    echo "Usage: $0 [options]"
+    echo ""
+    echo "Options:"
+    echo "  -h, --help                     Show this help message"
+    echo "      --scikit-learn-version    Scikit-learn version to test (default: 1.5.2)"
+    echo "  -p, --path                    Path to scikit-learn source (default: ./scikit-learn)"
+    exit 1
+}
+
+# Parse command line arguments
+while [[ $# -gt 0 ]]; do
+    case $1 in
+        -h|--help)
+            usage
+            ;;
+        --scikit-learn-version)
+            SCIKIT_LEARN_VERSION="$2"
+            shift 2
+            ;;
+        -p|--path)
+            SCIKIT_LEARN_SRC_PATH="$2"
+            shift 2
+            ;;
+        *)
+            echo "Unknown option: $1"
+            usage
+            ;;
+    esac
+done
+
+set -eu
+
+SCIKIT_LEARN_VERSION="${SCIKIT_LEARN_VERSION:-1.5.2}"
path: ci/accel/scikit-learn-tests/build.sh
user: login: bdice
body: I think a more natural default would be `main`, and then pin to `1.5.2` where this script is called (rather than hardcoding the version in this script itself).
created_at: 2025-03-10T21:02:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,57 @@
+#!/bin/bash
+# Copyright (c) 2025, NVIDIA CORPORATION.
+
+usage() {
+    echo "Usage: $0 [options]"
+    echo ""
+    echo "Options:"
+    echo "  -h, --help                     Show this help message"
+    echo "      --scikit-learn-version    Scikit-learn version to test (default: 1.5.2)"
+    echo "  -p, --path                    Path to scikit-learn source (default: ./scikit-learn)"
+    exit 1
+}
+
+# Parse command line arguments
+while [[ $# -gt 0 ]]; do
+    case $1 in
+        -h|--help)
+            usage
+            ;;
+        --scikit-learn-version)
+            SCIKIT_LEARN_VERSION="$2"
+            shift 2
+            ;;
+        -p|--path)
+            SCIKIT_LEARN_SRC_PATH="$2"
+            shift 2
+            ;;
+        *)
+            echo "Unknown option: $1"
+            usage
+            ;;
+    esac
+done
+
+set -eu
+
+SCIKIT_LEARN_VERSION="${SCIKIT_LEARN_VERSION:-1.5.2}"
+SCIKIT_LEARN_SRC_PATH="${SCIKIT_LEARN_SRC_PATH:-./scikit-learn}"
+
+# Clone scikit-learn if not already present and checkout version
+if [ ! -d "${SCIKIT_LEARN_SRC_PATH}" ]; then
+    git clone --depth 1 https://github.com/scikit-learn/scikit-learn.git ${SCIKIT_LEARN_SRC_PATH}
+fi
+
+cd ${SCIKIT_LEARN_SRC_PATH}
path: ci/accel/scikit-learn-tests/build.sh
user: login: bdice
body: Suggestion: Often in RAPIDS we use `pushd` and `popd` instead of `cd` so that CI scripts are a little more composable / less stateful.
created_at: 2025-03-10T21:05:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,59 @@
+# scikit-learn Acceleration Tests
path: ci/accel/scikit-learn-tests/README.md
user: login: jcrist
body: General comment on method here - you don't need a local copy of sklearn's git source to run the sklearn tests, they're packaged as part of sklearn itself. In any environment with `cuml` and `sklearn` both installed you can run the sklearn tests as:

```
py.test -p cuml.accel --pyargs sklearn  # or `sklearn.foo.tests.bar` for a specific test file
```

I have a slight preference for writing our test setup to use this style. This better isolates the installation of `sklearn` (and selection of version) from the test runner, and also avoids the need for extra state like `SCIKIT_LEARN_SOURCE_PATH` and friends.
created_at: 2025-03-11T19:40:23Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,59 @@
+# scikit-learn Acceleration Tests
path: ci/accel/scikit-learn-tests/README.md
user: login: csadorf
body: Ah nice, I wasn't aware of that. I'll update that.
created_at: 2025-03-11T20:09:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,59 @@
+# scikit-learn Acceleration Tests
path: ci/accel/scikit-learn-tests/README.md
user: login: csadorf
body: Adopted in https://github.com/rapidsai/cuml/pull/6377/commits/32a2a33abf2ead8a41ea4d94750434df95875106 .
created_at: 2025-03-12T16:06:10Z
author_association: CONTRIBUTOR

PR comments:
user: login: copy-pr-bot[bot]
created_at: 2025-02-27T22:00:22Z
body: Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message [here](https://docs.gha-runners.nvidia.com/cpr/auto-sync).

author_association: NONE
user: login: csadorf
created_at: 2025-02-27T22:03:26Z
body: /ok to test
author_association: CONTRIBUTOR
user: login: csadorf
created_at: 2025-02-27T22:41:59Z
body: /ok to test
author_association: CONTRIBUTOR
user: login: csadorf
created_at: 2025-02-28T16:26:09Z
body: /ok to test
author_association: CONTRIBUTOR
user: login: csadorf
created_at: 2025-03-05T17:32:04Z
body: /ok to test
author_association: CONTRIBUTOR
user: login: csadorf
created_at: 2025-03-07T14:55:09Z
body: The scikit-learn test suite takes about 30-45mins to complete. The job log is about 30MB long (~250k lines). We have a pass rate of about 88%. That's less than I had measured before, will have to investigate the cause for that. I am going to set the required pass rate at 80% for now.

@bdice This is ready for review.
author_association: CONTRIBUTOR
user: login: csadorf
created_at: 2025-03-13T18:34:58Z
body: /merge
author_association: CONTRIBUTOR