Repository: rapidsai/cucim (language: python)

Issue:
[BUG] fix isolated test failure in `test_cache_hit_miss`
**Describe the bug**

When working on wheel builds in #619, it was observed that a single test case, `test_cache_hit_miss` would fail when running the wheel tests. Temporarily, this test case is not being run because `imagecodecs` is not installed as part of the test dependencies during wheel testing.  Before release 23.12.00 we should enable install of `imagecodecs` and `openslide-python` again so that all `cucim.clara` submodule test cases get run.


build wheels on CI
This MR includes the commits from #617, so that one should be merged first.

The purpose of this MR is to build and test wheels on RAPIDS CI infrastructure. Previously, cuCIM was using a custom manual wheel building process based on running:
```
./run build_python_package
```
which used `dockcross`/`cibuildwheel`. We would then manually upload the resulting wheel to PyPI using `twine`.

The changes in this MR are to enable automated wheel builds for future RAPIDS releases in a similar way to other RAPIDS projects.

### Misc issues/questions for discussion
- The `build_wheels.sh` script installs the YASM and the openslide libraries via system package managers. This is currently hard-coded in that script rather than using the `dependencies.yaml` mechanism being used elsewhere. Is this reasonable or is there a better way?
- Following cuML's example, I added a separate `ci/release/apply_wheel_modifications.sh` script that updates the pyproject.toml as needed (to fixup version string and the CUDA version strings in the cucim package name and CuPy dependency)
- I updated the cuCIM test suite to make a handful of tests that relied on `imagecodecs` to be skipped if it is not installed. This is because I did not find a pre-compiled wheel for it on aarch64 and when it tried to compile from source instead, it failed due to various missing system packages needed for the compilation.
- Unlike most RAPIDS projects, which seem to use scikit-build, this project performs a separate CMake build via `run build_local` commands and copies the C++ libraries into the Python source tree. The `pyproject.toml` then just does a standard pure-python build, but will package the previously compiled libraries along with the other files.
- The wheel tests are currently taking approximately 35-50 minutes to run. It should be possible to reduce that time by an order of magnitude if the CuPy kernels are cached across runs. Does any RAPIDS projects cache the compiled CuPy kernels in this way? (cached kernels end up in `~/.cupy/kernel_cache/` by default, although an alternative location can be specified via `CUPY_CACHE_DIR`) 



Discussion / hints:

I think this recent failure will be fixed by merging #623:
https://github.com/rapidsai/cucim/actions/runs/6711052263/job/18238149350?pr=619

In discussion with Vyas offline, we've agreed to go ahead and merge. We can follow up on anything else in subsequent PRs/issues
Thanks Greg for the PR and everyone for the reviews! 🙏
/merge

Fix patch:
diff --git a/dependencies.yaml b/dependencies.yaml
index b7ceeef02..3e7732126 100644
--- a/dependencies.yaml
+++ b/dependencies.yaml
@@ -262,10 +262,9 @@ dependencies:
             - opencv-python-headless>=4.6
       - output_types: [requirements, pyproject]
         packages:
-          # temporarily remove imagecodecs / openslide-python from wheel tests
-          # # skip packages on arm64 that don't provide a wheel
-          # - imagecodecs>=2021.6.8; platform_machine=='x86_64'
-          # - openslide-python>=1.1.2; platform_machine=='x86_64'
+          # skip packages on arm64 that don't provide a wheel
+          - imagecodecs>=2021.6.8; platform_machine=='x86_64'
+          - openslide-python>=1.1.2; platform_machine=='x86_64'
           - matplotlib
           - opencv-python-headless>=4.6
           - click
diff --git a/python/cucim/pyproject.toml b/python/cucim/pyproject.toml
index 76695d5a9..880b1a06c 100644
--- a/python/cucim/pyproject.toml
+++ b/python/cucim/pyproject.toml
@@ -62,8 +62,10 @@ Tracker = "https://github.com/rapidsai/cucim/issues"
 test = [
     "GPUtil>=1.4.0",
     "click",
+    "imagecodecs>=2021.6.8; platform_machine=='x86_64'",
     "matplotlib",
     "opencv-python-headless>=4.6",
+    "openslide-python>=1.1.2; platform_machine=='x86_64'",
     "pooch>=1.6.0",
     "psutil>=5.8.0",
     "pytest-cov>=2.12.1",


Test patch:
diff --git a/ci/test_wheel.sh b/ci/test_wheel.sh
index 2fe2c1fb9..81f7f53e5 100755
--- a/ci/test_wheel.sh
+++ b/ci/test_wheel.sh
@@ -9,10 +9,21 @@ RAPIDS_PY_WHEEL_NAME="cucim_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-fro
 # echo to expand wildcard before adding `[extra]` requires for pip
 python -m pip install $(echo ./dist/cucim*.whl)[test]
 
+CUDA_MAJOR_VERSION=${RAPIDS_CUDA_VERSION:0:2}
+
 # Run smoke tests for aarch64 pull requests
 if [[ "$(arch)" == "aarch64" && ${RAPIDS_BUILD_TYPE} == "pull-request" ]]; then
     python ./ci/wheel_smoke_test.py
 else
-    # TODO: revisit enabling imagecodecs package during testing
-    python -m pytest ./python/cucim
+
+    DEBIAN_FRONTEND=noninteractive apt update
+    DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends libopenslide0
+
+    if [[ ${CUDA_MAJOR_VERSION} == "11" ]]; then
+        # Omit I/O-related tests in ./python/cucim/tests due to known CUDA bug
+        # with dynamic loading of libcufile.
+        python -m pytest ./python/cucim/src/
+    else
+        python -m pytest ./python/cucim
+    fi
 fi
diff --git a/python/cucim/tests/unit/clara/test_image_cache.py b/python/cucim/tests/unit/clara/test_image_cache.py
index 1b8987765..258e01da4 100644
--- a/python/cucim/tests/unit/clara/test_image_cache.py
+++ b/python/cucim/tests/unit/clara/test_image_cache.py
@@ -196,6 +196,7 @@ def test_reserve_more_cache_memory():
     assert cache.miss_count == 0
 
 
+@pytest.mark.skip(reason="currently fails (gh-626)")
 def test_cache_hit_miss(testimg_tiff_stripe_32x24_16_jpeg):
     from cucim import CuImage
     from cucim.clara.cache import preferred_memory_capacity