Repository: rapidsai/cugraph
Pull request #5027: Use pytest-xdist for parallel testing
Author: login: bdice
Labels: name: improvement
description: Improvement / enhancement to an existing function
name: non-breaking
description: Non-breaking change
name: python
name: ci
Created: 2025-04-11T17:56:32Z | Merged: 2025-05-14T14:10:40Z

We use `pytest-xdist` in almost every RAPIDS repository but haven't enabled it for cuGraph yet. Test times are long, so we should try to parallelize Python tests.

The [last successful CI run](https://github.com/rapidsai/cugraph/actions/runs/14391805813/job/40361124281#step:9:8233) took a long time:
```
3427 passed, 905 skipped, 51 deselected, 833 warnings in 5767.53s (1:36:07)
```

I successfully ran with `--numprocesses=8 --dist=worksteal` on my local machine:
```
3456 passed, 909 skipped, 1186 warnings in 881.19s (0:14:41)
```
and [CI got similar results](https://github.com/rapidsai/cugraph/actions/runs/14409723669/job/40415873062?pr=5027#step:9:13339):
```
3427 passed, 905 skipped, 887 warnings in 901.83s (0:15:01)
```

This suggests that there isn't too much time lost in the overhead of parallelizing the test jobs -- it's about 6.4x faster overall.

Our lowest-memory CI node has 24 GB of GPU memory. There are some tests that appear to spawn multiple processes (probably dask or MG tests) and the highest overall GPU memory usage I saw was ~18 GB -- still safely below the cap of 24 GB.

The maximum memory usage I saw was ~1830 MB for any given worker. I saw 3 workers at ~1830 MB and 5 workers at ~450-600 MB for most of the test time. The memory usage seemed to be monotonically increasing for each worker through the test run, which makes me think that resources (perhaps large graphs from datasets?) are not being cleaned up after they are used for a test.


Diff:
diff --git a/ci/test_python.sh b/ci/test_python.sh
index 61dc19a224b..f15d37a7a52 100755
--- a/ci/test_python.sh
+++ b/ci/test_python.sh
@@ -42,6 +42,9 @@ export LD_PRELOAD="${CONDA_PREFIX}/lib/libgomp.so.1"
 # RAPIDS_DATASET_ROOT_DIR is used by test scripts
 RAPIDS_DATASET_ROOT_DIR="$(realpath datasets)"
 export RAPIDS_DATASET_ROOT_DIR
+pushd "${RAPIDS_DATASET_ROOT_DIR}"
+./get_test_data.sh --subset
+popd
 
 EXITCODE=0
 trap "EXITCODE=1" ERR
@@ -51,6 +54,8 @@ rapids-logger "pytest pylibcugraph"
 ./ci/run_pylibcugraph_pytests.sh \
   --verbose \
   --junitxml="${RAPIDS_TESTS_DIR}/junit-pylibcugraph.xml" \
+  --numprocesses=8 \
+  --dist=worksteal \
   --cov-config=../../.coveragerc \
   --cov=pylibcugraph \
   --cov-report=xml:"${RAPIDS_COVERAGE_DIR}/pylibcugraph-coverage.xml" \
@@ -67,10 +72,39 @@ rapids-logger "pytest pylibcugraph"
 #
 # FIXME: TEMPORARILY disable MG PropertyGraph tests (experimental) tests and
 # bulk sampler IO tests (hangs in CI)
-rapids-logger "pytest cugraph"
+rapids-logger "pytest cugraph (not mg, with xdist)"
 ./ci/run_cugraph_pytests.sh \
   --verbose \
   --junitxml="${RAPIDS_TESTS_DIR}/junit-cugraph.xml" \
+  --numprocesses=8 \
+  --dist=worksteal \
+  -m "not mg" \
+  -k "not test_dataset and not test_bulk_sampler and not test_create_undirected_graph_from_asymmetric_adj_list and not test_uniform_neighbor_sample" \
+  --cov-config=../../.coveragerc \
+  --cov=cugraph \
+  --cov-report=xml:"${RAPIDS_COVERAGE_DIR}/cugraph-coverage.xml" \
+  --cov-report=term
+
+# Some tests fail with pytest-xdist enabled.
+# See https://github.com/rapidsai/cugraph/issues/5048
+rapids-logger "pytest cugraph (not mg, without xdist)"
+./ci/run_cugraph_pytests.sh \
+  --verbose \
+  --junitxml="${RAPIDS_TESTS_DIR}/junit-cugraph.xml" \
+  --numprocesses=1 \
+  --dist=worksteal \
+  -m "not mg" \
+  -k "not test_dataset and (test_bulk_sampler or test_create_undirected_graph_from_asymmetric_adj_list or test_uniform_neighbor_sample)" \
+  --cov-config=../../.coveragerc \
+  --cov=cugraph \
+  --cov-report=xml:"${RAPIDS_COVERAGE_DIR}/cugraph-coverage.xml" \
+  --cov-report=term
+
+rapids-logger "pytest cugraph (mg)"
+./ci/run_cugraph_pytests.sh \
+  --verbose \
+  --junitxml="${RAPIDS_TESTS_DIR}/junit-cugraph.xml" \
+  -m "mg" \
   --cov-config=../../.coveragerc \
   --cov=cugraph \
   --cov-report=xml:"${RAPIDS_COVERAGE_DIR}/cugraph-coverage.xml" \
diff --git a/dependencies.yaml b/dependencies.yaml
index 1c3fdc613d4..4769fd95ce3 100755
--- a/dependencies.yaml
+++ b/dependencies.yaml
@@ -439,7 +439,7 @@ dependencies:
           - cython>=3.0.0,<3.1.0a0
   python_run_cugraph:
     common:
-      - output_types: [conda, pyproject]
+      - output_types: [conda, pyproject, requirements]
         packages:
           - &dask rapids-dask-dependency==25.6.*,>=0.0.0a0
           - &numba numba>=0.59.1,<0.62.0a0
@@ -450,7 +450,7 @@ dependencies:
           - fsspec>=0.6.0
           - requests
           - nccl>=2.19
-      - output_types: pyproject
+      - output_types: [pyproject, requirements]
         packages:
             # cudf uses fsspec but is protocol independent. cugraph
             # dataset APIs require [http] extras for use with cudf.
@@ -472,12 +472,12 @@ dependencies:
           - *numpy
   python_run_cugraph_service_client:
     common:
-      - output_types: [conda, pyproject]
+      - output_types: [conda, pyproject, requirements]
         packages:
           - &thrift thriftpy2>=0.4.15,!=0.5.0,!=0.5.1
   python_run_cugraph_service_server:
     common:
-      - output_types: [conda, pyproject]
+      - output_types: [conda, pyproject, requirements]
         packages:
           - *dask
           - *numba
@@ -497,7 +497,7 @@ dependencies:
           - notebook>=0.5.0
   test_python_common:
     common:
-      - output_types: [conda, pyproject]
+      - output_types: [conda, pyproject, requirements]
         packages:
           - &pandas pandas
           - pytest
@@ -507,7 +507,7 @@ dependencies:
           - scipy
   test_python_cugraph:
     common:
-      - output_types: [conda, pyproject]
+      - output_types: [conda, pyproject, requirements]
         packages:
           - certifi
           - networkx>=2.5.1
diff --git a/python/cugraph/cugraph/tests/utils/test_dataset.py b/python/cugraph/cugraph/tests/utils/test_dataset.py
index 9895eb61c82..751388abb7f 100644
--- a/python/cugraph/cugraph/tests/utils/test_dataset.py
+++ b/python/cugraph/cugraph/tests/utils/test_dataset.py
@@ -1,4 +1,4 @@
-# Copyright (c) 2022-2024, NVIDIA CORPORATION.
+# Copyright (c) 2022-2025, NVIDIA CORPORATION.
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
@@ -81,6 +81,8 @@ def setup(tmpdir):
         dataset.unload()
     gc.collect()
 
+    datasets.set_download_dir(None)
+
 
 ###############################################################################
 # Helpers
@@ -151,6 +153,7 @@ def test_set_download_dir():
     assert datasets.get_download_dir() == Path(tmpd.name).absolute()
 
     tmpd.cleanup()
+    datasets.set_download_dir(None)
 
 
 @pytest.mark.parametrize("dataset", ALL_DATASETS)
@@ -253,6 +256,7 @@ def test_get_path(dataset):
 
     assert dataset.get_path().is_file()
     tmpd.cleanup()
+    datasets.set_download_dir(None)
 
 
 @pytest.mark.parametrize("dataset", WEIGHTED_DATASETS)
diff --git a/python/cugraph/cugraph/tests/utils/test_resultset.py b/python/cugraph/cugraph/tests/utils/test_resultset.py
index 5c2298bedb7..1b65344de3a 100644
--- a/python/cugraph/cugraph/tests/utils/test_resultset.py
+++ b/python/cugraph/cugraph/tests/utils/test_resultset.py
@@ -1,4 +1,4 @@
-# Copyright (c) 2023, NVIDIA CORPORATION.
+# Copyright (c) 2023-2025, NVIDIA CORPORATION.
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
@@ -45,6 +45,7 @@ def test_load_resultset():
 
         assert "resultsets.tar.gz" in os.listdir(datasets_download_dir / "tests")
         assert "traversal_mappings.csv" in os.listdir(resultsets_download_dir)
+    set_download_dir(None)
 
 
 def test_verify_resultset_load():


Reviews:
user: login: bdice
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T19:28:17Z
user: login: rlratzel
body: LGTM, thanks!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T20:30:19Z
user: login: jakirkham
body: Thanks Bradley! 🙏

This looks great 😄

Would it make sense to roll this out to more projects?
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-11T21:07:46Z
user: login: bdice
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-12T11:05:16Z
user: login: bdice
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-12T11:05:33Z
user: login: bdice
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-12T11:06:23Z
user: login: jameslamb
body: Giving you a `ci-codeowners` approval. I support using `pytest-xdist` here as we do in other repos.

Approving so you don't need a re-review, but of course please do revert all the commented-out stuff you're using for testing.

And can you please add `requirements` to the `output_types` here, so `pytest-xdist` will flow into the RAPIDS unified devcontainers test environment?

https://github.com/rapidsai/cugraph/blob/f4d530d873e15973db557cb4b24ed6afa882b8f4/dependencies.yaml#L492-L501

It's probably there anyway from other RAPIDS projects, but would be good to include it from this repo for completeness, and in case this repo ever adds pinning on `pytest-xdist` that is stricter than other RAPIDS repos.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-15T19:18:16Z
user: login: jameslamb
body: re-approving for `packaging-codeowners`
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-15T20:11:10Z
user: login: bdice
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T22:14:22Z
user: login: bdice
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T14:12:35Z

Review comments:
diff_hunk: @@ -83,6 +87,8 @@ rapids-logger "pytest cugraph-service (single GPU)"
 ./ci/run_cugraph_service_pytests.sh \
   --verbose \
   --junitxml="${RAPIDS_TESTS_DIR}/junit-cugraph-service.xml" \
+  --numprocesses=8 \
+  --dist=worksteal \
path: ci/test_python.sh
user: login: bdice
body: These fail. I'll revert this.
```suggestion
```
created_at: 2025-04-11T19:28:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,9 +1,9 @@
 #!/bin/bash
-# Copyright (c) 2024, NVIDIA CORPORATION.
+# Copyright (c) 2024-2025, NVIDIA CORPORATION.
 
 set -euo pipefail
 
 # Support invoking run_cugraph_service_pytests.sh outside the script directory
 cd "$(dirname "$(realpath "${BASH_SOURCE[0]}")")"/../python/cugraph-service
 
-pytest --capture=no --cache-clear --benchmark-disable -k "not mg" "$@" tests
+pytest --capture=no --cache-clear --benchmark-disable -m "not mg" "$@" tests
path: ci/run_cugraph_service_pytests.sh
user: login: bdice
body: Maybe this intends to filter test names and not the `mg` mark? If so, we probably aren't using the mark correctly.
```suggestion
pytest --capture=no --cache-clear --benchmark-disable -k "not mg" "$@" tests
```
created_at: 2025-04-12T11:05:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,5 +1,5 @@
 #!/bin/bash
-# Copyright (c) 2024, NVIDIA CORPORATION.
+# Copyright (c) 2024-2025, NVIDIA CORPORATION.
path: ci/run_cugraph_service_pytests.sh
user: login: bdice
body: ```suggestion
# Copyright (c) 2024, NVIDIA CORPORATION.
```
created_at: 2025-04-12T11:05:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -66,10 +71,24 @@ rapids-logger "pytest pylibcugraph"
 #
 # FIXME: TEMPORARILY disable MG PropertyGraph tests (experimental) tests and
 # bulk sampler IO tests (hangs in CI)
-rapids-logger "pytest cugraph"
+rapids-logger "pytest cugraph (not mg)"
 ./ci/run_cugraph_pytests.sh \
   --verbose \
   --junitxml="${RAPIDS_TESTS_DIR}/junit-cugraph.xml" \
+  --numprocesses=8 \
+  --dist=worksteal \
+  -m "not mg" \
+  --cov-config=../../.coveragerc \
+  --cov=cugraph \
+  --cov-report=xml:"${RAPIDS_COVERAGE_DIR}/cugraph-coverage.xml" \
+  --cov-report=term
+
+
+rapids-logger "pytest cugraph (mg)"
path: ci/test_python.sh
user: login: bdice
body: Multi-GPU tests consume the bulk of the time... I'll try parallelizing this again in a later commit.
created_at: 2025-04-12T11:06:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,6 +42,9 @@ export LD_PRELOAD="${CONDA_PREFIX}/lib/libgomp.so.1"
 # RAPIDS_DATASET_ROOT_DIR is used by test scripts
 RAPIDS_DATASET_ROOT_DIR="$(realpath datasets)"
 export RAPIDS_DATASET_ROOT_DIR
+pushd "${RAPIDS_DATASET_ROOT_DIR}"
+./get_test_data.sh --subset
+popd
path: ci/test_python.sh
user: login: bdice
body: I don't know for sure if adding this step is necessary or helpful. After we verify that CI passes on the latest commit, we should try removing this before merging.
created_at: 2025-05-12T22:11:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -67,10 +72,37 @@ rapids-logger "pytest pylibcugraph"
 #
 # FIXME: TEMPORARILY disable MG PropertyGraph tests (experimental) tests and
 # bulk sampler IO tests (hangs in CI)
-rapids-logger "pytest cugraph"
+rapids-logger "pytest cugraph (not mg, with xdist)"
 ./ci/run_cugraph_pytests.sh \
   --verbose \
   --junitxml="${RAPIDS_TESTS_DIR}/junit-cugraph.xml" \
+  --numprocesses=8 \
+  --dist=worksteal \
+  -m "not mg" \
+  -k "not test_dataset and not test_bulk_sampler and not test_create_undirected_graph_from_asymmetric_adj_list and not test_uniform_neighbor_sample" \
+  --cov-config=../../.coveragerc \
+  --cov=cugraph \
+  --cov-report=xml:"${RAPIDS_COVERAGE_DIR}/cugraph-coverage.xml" \
+  --cov-report=term
+
+rapids-logger "pytest cugraph (not mg, without xdist)"
path: ci/test_python.sh
user: login: bdice
body: ```suggestion
# Some tests fail with pytest-xdist enabled.
# See https://github.com/rapidsai/cugraph/issues/5048
rapids-logger "pytest cugraph (not mg, without xdist)"
```
created_at: 2025-05-12T22:13:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -66,10 +71,24 @@ rapids-logger "pytest pylibcugraph"
 #
 # FIXME: TEMPORARILY disable MG PropertyGraph tests (experimental) tests and
 # bulk sampler IO tests (hangs in CI)
-rapids-logger "pytest cugraph"
+rapids-logger "pytest cugraph (not mg)"
 ./ci/run_cugraph_pytests.sh \
   --verbose \
   --junitxml="${RAPIDS_TESTS_DIR}/junit-cugraph.xml" \
+  --numprocesses=8 \
+  --dist=worksteal \
+  -m "not mg" \
+  --cov-config=../../.coveragerc \
+  --cov=cugraph \
+  --cov-report=xml:"${RAPIDS_COVERAGE_DIR}/cugraph-coverage.xml" \
+  --cov-report=term
+
+
+rapids-logger "pytest cugraph (mg)"
path: ci/test_python.sh
user: login: bdice
body: Punting on this for a later PR.
created_at: 2025-05-12T22:13:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -81,6 +81,8 @@ def setup(tmpdir):
         dataset.unload()
     gc.collect()
 
+    datasets.set_download_dir(None)
path: python/cugraph/cugraph/tests/utils/test_dataset.py
user: login: bdice
body: All these manual cleanups are error-prone (easy to forget). We need to adopt some kind of self-cleaning fixture or context manager instead!
created_at: 2025-05-12T22:14:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,6 +42,9 @@ export LD_PRELOAD="${CONDA_PREFIX}/lib/libgomp.so.1"
 # RAPIDS_DATASET_ROOT_DIR is used by test scripts
 RAPIDS_DATASET_ROOT_DIR="$(realpath datasets)"
 export RAPIDS_DATASET_ROOT_DIR
+pushd "${RAPIDS_DATASET_ROOT_DIR}"
+./get_test_data.sh --subset
+popd
path: ci/test_python.sh
user: login: bdice
body: Testing this in #5072.
created_at: 2025-05-14T14:12:35Z
author_association: CONTRIBUTOR

PR comments:
user: login: bdice
created_at: 2025-04-11T21:36:41Z
body: > Would it make sense to roll this out to more projects?

We have already rolled out `pytest-xdist` and `--dist=worksteal` to all other repositories. cuGraph has been needing this for a long time.
author_association: CONTRIBUTOR
user: login: bdice
created_at: 2025-04-11T21:48:41Z
body: It seems like there may be a couple issues.

- Dask/mg tests seem to fail when run with xdist. We should invoke those in a separate command. See ce2205c, 88d2b2c.
- Maybe race conditions in downloading datasets? It looks like some tests explicitly download the data files, and other tests rely on those files. We should download all files ahead of time. I'll see what happens after running again.
author_association: CONTRIBUTOR
user: login: bdice
created_at: 2025-04-14T15:55:54Z
body: @rlratzel Could you help diagnose these errors? I have been struggling to figure out the root cause. I can't reproduce it locally. I wonder if there's some logic in the cuGraph tests that is using temporary directories in a stateful way?

```
=========================== short test summary info ============================
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_tree[True] - FileNotFoundError: ['/tmp/tmpa2fhi385/small_tree.csv'] could not be resolved to any files
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_tree[False] - FileNotFoundError: ['/tmp/tmpa2fhi385/small_tree.csv'] could not be resolved to any files
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_exclude_sources_email_eu_core - RuntimeError: The datafile /tmp/tmpa2fhi385/email-Eu-core.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_carry_over_sources_email_eu_core - RuntimeError: The datafile /tmp/tmpa2fhi385/email-Eu-core.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_deduplicate_sources_email_eu_core - RuntimeError: The datafile /tmp/tmpa2fhi385/email-Eu-core.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_renumber[hops0] - RuntimeError: The datafile /tmp/tmpa2fhi385/email-Eu-core.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_renumber[hops1] - RuntimeError: The datafile /tmp/tmpa2fhi385/email-Eu-core.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_renumber[hops2] - RuntimeError: The datafile /tmp/tmpa2fhi385/email-Eu-core.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_offset_renumber[hops0] - RuntimeError: The datafile /tmp/tmpa2fhi385/email-Eu-core.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_offset_renumber[hops1] - RuntimeError: The datafile /tmp/tmpa2fhi385/email-Eu-core.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_offset_renumber[hops2] - RuntimeError: The datafile /tmp/tmpa2fhi385/email-Eu-core.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_csr_csc_global[62-hops0] - RuntimeError: The datafile /tmp/tmpa2fhi385/email-Eu-core.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_csr_csc_global[62-hops1] - RuntimeError: The datafile /tmp/tmpa2fhi385/email-Eu-core.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_csr_csc_global[62-hops2] - RuntimeError: The datafile /tmp/tmpa2fhi385/email-Eu-core.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_csr_csc_global[66-hops0] - RuntimeError: The datafile /tmp/tmpa2fhi385/email-Eu-core.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_csr_csc_global[66-hops1] - RuntimeError: The datafile /tmp/tmpa2fhi385/email-Eu-core.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_csr_csc_global[66-hops2] - RuntimeError: The datafile /tmp/tmpa2fhi385/email-Eu-core.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_csr_csc_global[68-hops0] - RuntimeError: The datafile /tmp/tmpa2fhi385/email-Eu-core.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_csr_csc_global[68-hops1] - RuntimeError: The datafile /tmp/tmpa2fhi385/email-Eu-core.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_csr_csc_global[68-hops2] - RuntimeError: The datafile /tmp/tmpa2fhi385/email-Eu-core.csv does not exist. Try setting download=True to download the datafile
FAILED tests/nx/test_nx_convert.py::test_networkx_compatibility[graph_file0] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzcu3j87m/dolphins.csv'
FAILED tests/nx/test_nx_convert.py::test_networkx_compatibility[graph_file2] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzcu3j87m/karate-disjoint.csv'
FAILED tests/nx/test_nx_convert.py::test_nx_convert_undirected[graph_file0] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzcu3j87m/dolphins.csv'
FAILED tests/nx/test_nx_convert.py::test_nx_convert_undirected[graph_file2] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzcu3j87m/karate-disjoint.csv'
FAILED tests/nx/test_nx_convert.py::test_nx_convert_directed[graph_file0] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzcu3j87m/dolphins.csv'
FAILED tests/nx/test_nx_convert.py::test_nx_convert_directed[graph_file2] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzcu3j87m/karate-disjoint.csv'
FAILED tests/nx/test_nx_convert.py::test_nx_convert_weighted[graph_file0] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzcu3j87m/dolphins.csv'
FAILED tests/nx/test_nx_convert.py::test_nx_convert_weighted[graph_file2] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzcu3j87m/karate-disjoint.csv'
FAILED tests/nx/test_nx_convert.py::test_nx_convert_multicol[graph_file0] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzcu3j87m/dolphins.csv'
FAILED tests/nx/test_nx_convert.py::test_nx_convert_multicol[graph_file2] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzcu3j87m/karate-disjoint.csv'
FAILED tests/sampling/test_biased_random_walks.py::test_biased_random_walks_invalid_max_dept[None-False-graph_file1] - RuntimeError: The datafile /tmp/tmpzcu3j87m/dolphins.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_biased_random_walks.py::test_biased_random_walks_invalid_max_dept[None-False-graph_file2] - RuntimeError: The datafile /tmp/tmpzcu3j87m/polbooks.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_biased_random_walks.py::test_biased_random_walks_invalid_max_dept[None-True-graph_file1] - RuntimeError: The datafile /tmp/tmpzcu3j87m/dolphins.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_biased_random_walks.py::test_biased_random_walks_invalid_max_dept[None-True-graph_file2] - RuntimeError: The datafile /tmp/tmpzcu3j87m/polbooks.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_biased_random_walks.py::test_biased_random_walks[False-graph_file1] - RuntimeError: The datafile /tmp/tmpzcu3j87m/dolphins.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_biased_random_walks.py::test_biased_random_walks[False-graph_file2] - RuntimeError: The datafile /tmp/tmpzcu3j87m/polbooks.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_biased_random_walks.py::test_biased_random_walks[True-graph_file1] - RuntimeError: The datafile /tmp/tmpzcu3j87m/dolphins.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_biased_random_walks.py::test_biased_random_walks[True-graph_file2] - RuntimeError: The datafile /tmp/tmpzcu3j87m/polbooks.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_biased_random_walks.py::test_biased_random_walks_multi_column_seeds[False-graph_file1] - RuntimeError: The datafile /tmp/tmpzcu3j87m/dolphins.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_biased_random_walks.py::test_biased_random_walks_multi_column_seeds[False-graph_file2] - RuntimeError: The datafile /tmp/tmpzcu3j87m/polbooks.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_biased_random_walks.py::test_biased_random_walks_multi_column_seeds[True-graph_file1] - RuntimeError: The datafile /tmp/tmpzcu3j87m/dolphins.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_biased_random_walks.py::test_biased_random_walks_multi_column_seeds[True-graph_file2] - RuntimeError: The datafile /tmp/tmpzcu3j87m/polbooks.csv does not exist. Try setting download=True to download the datafile
FAILED tests/sampling/test_bulk_sampler.py::test_bulk_sampler_csr - RuntimeError: The datafile /tmp/tmpzcu3j87m/email-Eu-core.csv does not exist. Try setting download=True to download the datafile
FAILED tests/community/test_ecg.py::test_ecg_clustering[42-1.0-1e-07-20-16-0.15-dataset1] - RuntimeError: The datafile /tmp/tmpzcu3j87m/dolphins.csv does not exist. Try setting download=True to download the datafile
FAILED tests/community/test_ecg.py::test_ecg_clustering[42-1.0-1e-07-20-32-0.05-dataset1] - RuntimeError: The datafile /tmp/tmpzcu3j87m/dolphins.csv does not exist. Try setting download=True to download the datafile
FAILED tests/community/test_ecg.py::test_ecg_clustering[42-1.0-1e-07-20-32-0.15-dataset1] - RuntimeError: The datafile /tmp/tmpzcu3j87m/dolphins.csv does not exist. Try setting download=True to download the datafile
FAILED tests/community/test_ecg.py::test_ecg_clustering_nx[0-0.95-1e-06-10-16-0.05-dataset1] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzcu3j87m/dolphins.csv'
FAILED tests/community/test_ecg.py::test_ecg_clustering_nx[0-0.95-1e-06-10-16-0.15-dataset1] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzcu3j87m/dolphins.csv'
FAILED tests/community/test_ecg.py::test_ecg_clustering_nx[0-0.95-1e-06-10-32-0.05-dataset1] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzcu3j87m/dolphins.csv'
FAILED tests/community/test_ecg.py::test_ecg_clustering_nx[0-0.95-1e-06-10-32-0.15-dataset1] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzcu3j87m/dolphins.csv'
FAILED tests/community/test_ecg.py::test_ecg_clustering_nx[0-0.95-1e-06-20-16-0.05-dataset1] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzcu3j87m/dolphins.csv'
FAILED tests/community/test_ecg.py::test_ecg_clustering_nx[0-0.95-1e-06-20-16-0.15-dataset1] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzcu3j87m/dolphins.csv'
FAILED tests/community/test_ecg.py::test_ecg_clustering_nx[0-0.95-1e-06-20-32-0.05-dataset1] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzcu3j87m/dolphins.csv'
FAILED tests/community/test_ecg.py::test_ecg_clustering_nx[0-0.95-1e-06-20-32-0.15-dataset1] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzcu3j87m/dolphins.csv'
FAILED tests/community/test_ecg.py::test_ecg_clustering_nx[0-0.95-1e-07-10-16-0.05-dataset1] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzcu3j87m/dolphins.csv'
ERROR tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_simple[graph_file:email_Eu_core-directed:True-with_replacement:False-indices_type:int32] - FileNotFoundError: ['/tmp/tmpa2fhi385/email-Eu-core.csv'] could not be resolved to any files
ERROR tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_simple[graph_file:email_Eu_core-directed:True-with_replacement:False-indices_type:float32] - FileNotFoundError: ['/tmp/tmpa2fhi385/email-Eu-core.csv'] could not be resolved to any files
ERROR tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_simple[graph_file:email_Eu_core-directed:True-with_replacement:True-indices_type:int32] - FileNotFoundError: ['/tmp/tmpa2fhi385/email-Eu-core.csv'] could not be resolved to any files
ERROR tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_simple[graph_file:email_Eu_core-directed:True-with_replacement:True-indices_type:float32] - FileNotFoundError: ['/tmp/tmpa2fhi385/email-Eu-core.csv'] could not be resolved to any files
ERROR tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_simple[graph_file:email_Eu_core-directed:False-with_replacement:False-indices_type:int32] - FileNotFoundError: ['/tmp/tmpa2fhi385/email-Eu-core.csv'] could not be resolved to any files
ERROR tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_simple[graph_file:email_Eu_core-directed:False-with_replacement:False-indices_type:float32] - FileNotFoundError: ['/tmp/tmpa2fhi385/email-Eu-core.csv'] could not be resolved to any files
ERROR tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_simple[graph_file:email_Eu_core-directed:False-with_replacement:True-indices_type:int32] - FileNotFoundError: ['/tmp/tmpa2fhi385/email-Eu-core.csv'] could not be resolved to any files
ERROR tests/sampling/test_uniform_neighbor_sample.py::test_uniform_neighbor_sample_simple[graph_file:email_Eu_core-directed:False-with_replacement:True-indices_type:float32] - FileNotFoundError: ['/tmp/tmpa2fhi385/email-Eu-core.csv'] could not be resolved to any files
= 55 failed, 1977 passed, 823 skipped, 527 warnings, 8 errors in 614.09s (0:10:14) =
```
author_association: CONTRIBUTOR
user: login: bdice
created_at: 2025-05-07T19:00:16Z
body: @rlratzel What do you think of merging this as-is? It seems to shave off some time in tests. We still need to try to make multi-GPU tests run in parallel. We also need to investigate why some tests aren't working with pytest-xdist -- probably manipulation of the datasets directory that isn't being cleaned up. But I think this is a good starting point. If you agree I will re-enable the disabled CI jobs and we can merge.

Comparing a run of [this PR](https://github.com/rapidsai/cugraph/actions/runs/14871264875/job/41761196653?pr=5027) to [the most recently merged PR](https://github.com/rapidsai/cugraph/actions/runs/14869304034/job/41757352691), I see:
- conda-python-tests / 11.4.3, 3.10, amd64, rockylinux8, l4, earliest-driver, oldest-deps
  - 1h 20min vs. 1h 44min (faster by 24 min)
- conda-python-tests / 11.8.0, 3.10, amd64, rockylinux8, l4, earliest-driver, oldest-deps
  - 1h 17min vs. 1h 48min (faster by 31 min)
- conda-python-tests / 12.8.0, 3.12, amd64, ubuntu24.04, h100, latest-driver, latest-deps
  - 1h 50min vs. 1h 58min (faster by 8 min)
- conda-python-tests / 12.0.1, 3.11, arm64, ubuntu20.04, a100, latest-driver, latest-deps
  - 1h 37min vs. 2h 9min (faster by 32 min)
author_association: CONTRIBUTOR
user: login: nv-rliu
created_at: 2025-05-12T19:15:56Z
body: Hi @bdice stepping in here to help with the efforts adding `xdist` to cuGraph's CI. Thanks for all the work. We should be able to merge this as-is to propagate the speedups to other PRs while we work on investigating why some tests aren't cooperating with `xdist`. 
author_association: CONTRIBUTOR
user: login: bdice
created_at: 2025-05-14T14:09:42Z
body: @nv-rliu @rlratzel Can I get a review here? It cuts Python CI time by ~22%.
author_association: CONTRIBUTOR
user: login: bdice
created_at: 2025-05-14T14:10:30Z
body: Oh weird, it seems like the Python codeowners are still requested, though Rick has reviewed. Apologies for the ping. I'll go ahead and merge this. @nv-rliu -- it sounds like you wanted to pick up the follow-up items. Happy to help out where I can!
author_association: CONTRIBUTOR
user: login: bdice
created_at: 2025-05-14T14:10:35Z
body: /merge
author_association: CONTRIBUTOR
user: login: vyasr
created_at: 2025-05-14T18:41:00Z
body: This PR partially addresses https://github.com/rapidsai/cugraph/issues/3047 by parallelizing the tests that did not encounter the Dataset issues described there.
author_association: CONTRIBUTOR