Repository: rapidsai/kvikio (language: python)

Issue:
add wheel output
closes #250 

I'm not sure if it makes sense to split the wheel into one dedicated wheel for the shared library, and another for the python wrapper. That's what @vyasr has been doing on other libraries, like https://github.com/rapidsai/rmm/pull/1529. If it makes sense here, I'll add that.


Discussion / hints:
Yeah for consistency with the rest of RAPIDS using separate C++ and Python wheels is the way to go. We'll be able to leverage that by using the libkvikio wheel during the libcudf wheel build since libcudf needs libkvikio headers. You can see that in [your cudf PR](https://github.com/rapidsai/cudf/pull/15483) those are [currently being bundled into the libcudf wheel](https://github.com/rapidsai/cudf/actions/runs/8714266105/job/23904261719?pr=15483#step:9:2552) which is not ideal. Using the libkvikio wheel from this PR instead will fix that.
Error in wheel build:

```
2024-04-23T14:25:13.7290506Z   CMake Debug Log at /tmp/pip-build-env-c44ywnqk/overlay/lib/python3.11/site-packages/cmake/data/share/cmake-3.29/Modules/FindCUDAToolkit.cmake:1093 (find_library):
2024-04-23T14:25:13.7290723Z     find_library called with the following settings:
2024-04-23T14:25:13.7291226Z 
2024-04-23T14:25:13.7292425Z       VAR: CUDA_nvgraph_LIBRARY
2024-04-23T14:25:13.7293410Z       NAMES: "nvgraph"
2024-04-23T14:25:13.7294445Z       Documentation: Path to a library.
2024-04-23T14:25:13.7295560Z       Framework
2024-04-23T14:25:13.7296475Z         Only Search Frameworks: 0
2024-04-23T14:25:13.7297410Z         Search Frameworks Last: 0
2024-04-23T14:25:13.7298380Z         Search Frameworks First: 0
2024-04-23T14:25:13.7299296Z       AppBundle
2024-04-23T14:25:13.7300297Z         Only Search AppBundle: 0
2024-04-23T14:25:13.7301260Z         Search AppBundle Last: 0
2024-04-23T14:25:13.7302243Z         Search AppBundle First: 0
2024-04-23T14:25:13.7303230Z       CMAKE_FIND_USE_CMAKE_PATH: 1
2024-04-23T14:25:13.7304297Z       CMAKE_FIND_USE_CMAKE_ENVIRONMENT_PATH: 1
2024-04-23T14:25:13.7305256Z       CMAKE_FIND_USE_SYSTEM_ENVIRONMENT_PATH: 1
2024-04-23T14:25:13.7306202Z       CMAKE_FIND_USE_CMAKE_SYSTEM_PATH: 1
2024-04-23T14:25:13.7307195Z       CMAKE_FIND_USE_INSTALL_PREFIX: 1
2024-04-23T14:25:13.7308173Z 
2024-04-23T14:25:13.7309466Z     find_library considered the following locations:
2024-04-23T14:25:13.7310161Z 
2024-04-23T14:25:13.7311636Z       /pyenv/versions/3.11.9/lib/python3.11/site-packages/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7312856Z       /tmp/pip-build-env-c44ywnqk/overlay/lib/python3.11/site-packages/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7313631Z       /usr/local/cuda-11.8/targets/sbsa-linux/lib/stubs/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7314756Z       /usr/local/cuda-11.8/targets/sbsa-linux/lib/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7315623Z       /usr/local/cuda/targets/sbsa-linux/lib/stubs/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7316798Z       /usr/local/cuda/targets/sbsa-linux/lib/stubs/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7317635Z       /usr/local/cuda/targets/sbsa-linux/lib/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7318804Z       /tmp/pip-build-env-c44ywnqk/overlay/bin/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7319505Z       /pyenv/versions/3.11.9/bin/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7320537Z       /pyenv/libexec/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7321656Z       /pyenv/plugins/python-build/bin/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7322702Z       /pyenv/plugins/pyenv-virtualenv/bin/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7323693Z       /pyenv/plugins/pyenv-update/bin/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7324697Z       /pyenv/plugins/pyenv-doctor/bin/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7325540Z       /usr/share/Modules/bin/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7326723Z       /opt/rh/gcc-toolset-11/root/usr/bin/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7327383Z       /pyenv/bin/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7328500Z       /pyenv/shims/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7329497Z       /usr/local/cuda/bin/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7330462Z       /usr/local/sbin/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7331416Z       /usr/local/bin/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7332371Z       /usr/sbin/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7333400Z       /usr/bin/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7334331Z       /sbin/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7335402Z       /bin/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7336379Z       /usr/local/lib64/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7337336Z       /usr/local/lib/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7338301Z       /usr/local/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7339339Z       /usr/lib64/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7340462Z       /usr/lib/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7341307Z       /usr/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7342246Z       /lib64/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7343187Z       /lib/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7344763Z       /tmp/pip-build-env-c44ywnqk/overlay/lib/python3.11/site-packages/cmake/data/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7345173Z       /opt/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7346285Z       /usr/lib64/X11/libnvgraph(\.so|\.a)
2024-04-23T14:25:13.7347077Z 
2024-04-23T14:25:13.7348338Z     The item was not found.
```
Feel free to add this to the [devcontainer build job](https://github.com/rapidsai/kvikio/blob/7b0231c8241164c8a0511cd737e7098cb7ccea3e/.github/workflows/pr.yaml#L67-L70) to test the path adjustment in the devcontainers before merging:
```yaml
      build_command: |
        sudo yq -i '.repos[2].python[0].sub_dir = "python/kvikio"' /opt/rapids-build-utils/manifest.yaml;
        rapids-generate-scripts;
```
Youll need to add a line [like this](https://github.com/rapidsai/rmm/blob/branch-24.06/python/librmm/pyproject.toml#L55) indicating that the wheels built for libkvikio are compatible with any Python version. Without that, scikit-build assumes that the wheels are compiled against Python and are therefore only compatible with the minor version of Python that you built against (in this case, Python 3.11).
Finally have passing tests here. @vyasr please take one more pass on this.
For anyone new to this thread, the failures in the devcontainers builds are expected right now. The plan to proceed is to get approval from ops and kvikio code owners on this PR, then merge the related devcontainers PR (https://github.com/rapidsai/devcontainers/pull/283), then re-run CI here. Assuming all goes well with that re-run, this PR will be ready to merge.
Once we have approvals, the final steps to merge will be
1. Cleaning out the special logic in this PR for running devcontainer jobs without the upstream updates
2. Merging the devcontainer PR
3. Rerunning CI here to make sure everything passes

Fix patch:
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index be56a05348..8f318f43da 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -82,7 +82,7 @@ repos:
             ^CHANGELOG.md$
           )
   - repo: https://github.com/rapidsai/dependency-file-generator
-    rev: v1.8.0
+    rev: v1.13.11
     hooks:
       - id: rapids-dependency-file-generator
         args: ["--clean"]
diff --git a/build.sh b/build.sh
index fc4b1d6df1..3b790bf20a 100755
--- a/build.sh
+++ b/build.sh
@@ -153,5 +153,5 @@ if (( NUMARGS == 0 )) || hasArg kvikio; then
     echo "building kvikio..."
     cd ${REPODIR}/python
     SKBUILD_CMAKE_ARGS="-DCMAKE_PREFIX_PATH=${INSTALL_PREFIX};-DCMAKE_LIBRARY_PATH=${LIBKVIKIO_BUILD_DIR};${EXTRA_CMAKE_ARGS}" \
-        python -m pip install --no-build-isolation --no-deps .
+        python -m pip install --no-build-isolation --no-deps --config-settings rapidsai.disable-cuda=true .
 fi
diff --git a/ci/build_docs.sh b/ci/build_docs.sh
index 7bd14e4973..788b3517b8 100755
--- a/ci/build_docs.sh
+++ b/ci/build_docs.sh
@@ -8,7 +8,7 @@ rapids-logger "Create test conda environment"
 
 rapids-dependency-file-generator \
   --output conda \
-  --file_key docs \
+  --file-key docs \
   --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee env.yaml
 
 rapids-mamba-retry env create --yes -f env.yaml -n docs
diff --git a/ci/check_style.sh b/ci/check_style.sh
index d154742114..a17065c1f9 100755
--- a/ci/check_style.sh
+++ b/ci/check_style.sh
@@ -8,7 +8,7 @@ rapids-logger "Create checks conda environment"
 
 rapids-dependency-file-generator \
   --output conda \
-  --file_key checks \
+  --file-key checks \
   --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee env.yaml
 
 rapids-mamba-retry env create --yes -f env.yaml -n checks
diff --git a/conda/environments/all_cuda-118_arch-x86_64.yaml b/conda/environments/all_cuda-118_arch-x86_64.yaml
index 03c84b29a7..b283cb3388 100644
--- a/conda/environments/all_cuda-118_arch-x86_64.yaml
+++ b/conda/environments/all_cuda-118_arch-x86_64.yaml
@@ -30,6 +30,7 @@ dependencies:
 - pytest
 - pytest-cov
 - python>=3.9,<3.12
+- rapids-build-backend>=0.3.0,<0.4.0.dev0
 - scikit-build-core>=0.7.0
 - sphinx
 - sphinx-click
diff --git a/conda/environments/all_cuda-122_arch-x86_64.yaml b/conda/environments/all_cuda-122_arch-x86_64.yaml
index e1a8f31c5b..657e7b3e25 100644
--- a/conda/environments/all_cuda-122_arch-x86_64.yaml
+++ b/conda/environments/all_cuda-122_arch-x86_64.yaml
@@ -28,6 +28,7 @@ dependencies:
 - pytest
 - pytest-cov
 - python>=3.9,<3.12
+- rapids-build-backend>=0.3.0,<0.4.0.dev0
 - scikit-build-core>=0.7.0
 - sphinx
 - sphinx-click
diff --git a/conda/recipes/kvikio/meta.yaml b/conda/recipes/kvikio/meta.yaml
index 886334be5d..2f2e308ae7 100644
--- a/conda/recipes/kvikio/meta.yaml
+++ b/conda/recipes/kvikio/meta.yaml
@@ -61,6 +61,7 @@ requirements:
     {% endif %}
     - cuda-version ={{ cuda_version }}
     - nvcomp {{ nvcomp_version }}
+    - rapids-build-backend >=0.3.0,<0.4.0.dev0
     - scikit-build-core >=0.7.0
     - libkvikio ={{ version }}
   run:
diff --git a/dependencies.yaml b/dependencies.yaml
index 1e17c924f0..c04583a4c7 100644
--- a/dependencies.yaml
+++ b/dependencies.yaml
@@ -13,6 +13,7 @@ files:
       - depends_on_cupy
       - docs
       - py_version
+      - rapids_build_skbuild
       - run
       - test_python
   test_cpp:
@@ -41,6 +42,14 @@ files:
     pyproject_dir: python
     extras:
       table: build-system
+    includes:
+      - rapids_build_skbuild
+  py_rapids_build:
+    output: pyproject
+    pyproject_dir: python
+    extras:
+      table: tool.rapids-build-backend
+      key: requires
     includes:
       - build
   py_run:
@@ -76,10 +85,6 @@ dependencies:
         packages:
           - c-compiler
           - cxx-compiler
-          - scikit-build-core>=0.7.0
-      - output_types: [requirements, pyproject]
-        packages:
-          - scikit-build-core[pyproject]>=0.7.0
     specific:
       - output_types: conda
         matrices:
@@ -238,6 +243,17 @@ dependencies:
           - matrix:
             packages:
               - python>=3.9,<3.12
+  rapids_build_skbuild:
+    common:
+      - output_types: [conda, requirements, pyproject]
+        packages:
+          - rapids-build-backend>=0.3.0,<0.4.0.dev0
+      - output_types: conda
+        packages:
+          - scikit-build-core>=0.7.0
+      - output_types: [requirements, pyproject]
+        packages:
+          - scikit-build-core[pyproject]>=0.7.0
   run:
     common:
       - output_types: [conda, requirements, pyproject]
diff --git a/python/kvikio/__init__.py b/python/kvikio/__init__.py
index 5186df50f8..5e2704adfa 100644
--- a/python/kvikio/__init__.py
+++ b/python/kvikio/__init__.py
@@ -2,7 +2,7 @@
 # See file LICENSE for terms.
 
 from ._lib import libkvikio  # type: ignore
-from ._version import __version__  # noqa: F401
+from ._version import __git_commit__, __version__  # noqa: F401
 from .cufile import CuFile  # noqa: F401
 
 
diff --git a/python/kvikio/_version.py b/python/kvikio/_version.py
index 82fc478d5c..a5171f19f4 100644
--- a/python/kvikio/_version.py
+++ b/python/kvikio/_version.py
@@ -15,6 +15,16 @@
 import importlib.resources
 
 __version__ = (
-    importlib.resources.files("kvikio").joinpath("VERSION").read_text().strip()
+    importlib.resources.files(__package__).joinpath("VERSION").read_text().strip()
 )
-__git_commit__ = ""
+try:
+    __git_commit__ = (
+        importlib.resources.files(__package__)
+        .joinpath("GIT_COMMIT")
+        .read_text()
+        .strip()
+    )
+except FileNotFoundError:
+    __git_commit__ = ""
+
+__all__ = ["__git_commit__", "__version__"]
diff --git a/python/pyproject.toml b/python/pyproject.toml
index 743c832ece..c1b99ed583 100644
--- a/python/pyproject.toml
+++ b/python/pyproject.toml
@@ -2,11 +2,9 @@
 # See file LICENSE for terms.
 
 [build-system]
-build-backend = "scikit_build_core.build"
+build-backend = "rapids_build_backend.build"
 requires = [
-    "cmake>=3.26.4",
-    "cython>=3.0.0",
-    "ninja",
+    "rapids-build-backend>=0.3.0,<0.4.0.dev0",
     "scikit-build-core[pyproject]>=0.7.0",
 ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../dependencies.yaml and run `rapids-dependency-file-generator`.
 
@@ -109,6 +107,15 @@ ignore_missing_imports = true
 [project.entry-points."numcodecs.codecs"]
 nvcomp_batch = "kvikio.nvcomp_codec:NvCompBatchCodec"
 
+[tool.rapids-build-backend]
+build-backend = "scikit_build_core.build"
+dependencies-file = "../dependencies.yaml"
+requires = [
+    "cmake>=3.26.4",
+    "cython>=3.0.0",
+    "ninja",
+] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../dependencies.yaml and run `rapids-dependency-file-generator`.
+
 [tool.scikit-build]
 build-dir = "build/{wheel_tag}"
 cmake.build-type = "Release"


Test patch:
diff --git a/ci/test_cpp.sh b/ci/test_cpp.sh
index fdc6584a14..e7de3e68f2 100755
--- a/ci/test_cpp.sh
+++ b/ci/test_cpp.sh
@@ -8,7 +8,7 @@ set -euo pipefail
 rapids-logger "Generate C++ testing dependencies"
 rapids-dependency-file-generator \
   --output conda \
-  --file_key test_cpp \
+  --file-key test_cpp \
   --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch)" | tee env.yaml
 
 rapids-mamba-retry env create --yes -f env.yaml -n test
diff --git a/ci/test_python.sh b/ci/test_python.sh
index ecd2bea44e..fccbcb4728 100755
--- a/ci/test_python.sh
+++ b/ci/test_python.sh
@@ -11,7 +11,7 @@ cd "$(dirname "$(realpath "${BASH_SOURCE[0]}")")"/../
 rapids-logger "Generate Python testing dependencies"
 rapids-dependency-file-generator \
   --output conda \
-  --file_key test_python \
+  --file-key test_python \
   --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee env.yaml
 
 rapids-mamba-retry env create --yes -f env.yaml -n test
diff --git a/python/tests/test_version.py b/python/tests/test_version.py
new file mode 100644
index 0000000000..840d6624c9
--- /dev/null
+++ b/python/tests/test_version.py
@@ -0,0 +1,13 @@
+# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
+# See file LICENSE for terms.
+
+import kvikio
+
+
+def test_version_constants_are_populated():
+    # __git_commit__ will only be non-empty in a built distribution
+    assert isinstance(kvikio.__git_commit__, str)
+
+    # __version__ should always be non-empty
+    assert isinstance(kvikio.__version__, str)
+    assert len(kvikio.__version__) > 0