Repository: apache/incubator-gluten
Pull request #9333: [GLUTEN-9243][VL] Fix cuda docker image
Author: login: zhouyuan
Labels: name: BUILD
name: VELOX
name: INFRA
Created: 2025-04-15T10:34:11Z | Merged: 2025-04-18T15:02:15Z

## What changes were proposed in this pull request?

The image `ghcr.io/facebookincubator/velox-dev:adapters` is not avaiable on ARM
followup on https://github.com/apache/incubator-gluten/pull/9229

Signed-off-by: Yuan Zhou <yuan.zhou@ibm.com>

## How was this patch tested?
pass GHA




Diff:
diff --git a/.github/workflows/docker_image.yml b/.github/workflows/docker_image.yml
index 4e94e73e9d0a..3403ab762da6 100644
--- a/.github/workflows/docker_image.yml
+++ b/.github/workflows/docker_image.yml
@@ -60,23 +60,13 @@ jobs:
           push: true
           tags: ${{ env.DOCKERHUB_REPO }}:vcpkg-centos-7
 
-  build-vcpkg-centos-8:
+  build-centos-9-jdk8-cudf:
     if: ${{ startsWith(github.repository, 'apache/') }}
-    runs-on: ${{ matrix.os }}
-    strategy:
-      matrix:
-        os: [ ubuntu-latest, ubuntu-24.04-arm ]
+    runs-on: ubuntu-latest
 
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v2
-
-      - name: Docker meta
-        id: meta
-        uses: docker/metadata-action@v5
-        with:
-          images: ${{ env.DOCKERHUB_REPO }}
-          tags: vcpkg-centos-8
+        uses: actions/checkout@v4
 
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
@@ -87,30 +77,15 @@ jobs:
           username: ${{ secrets.DOCKERHUB_USER }}
           password: ${{ secrets.DOCKERHUB_TOKEN }}
 
-      - name: Build and push by digest
-        id: build
+      - name: Build and push Docker image
         uses: docker/build-push-action@v6
         with:
           context: .
-          file: dev/docker/Dockerfile.centos8-static-build
-          labels: ${{ steps.meta.outputs.labels }}
-          outputs: type=image,"name=${{ env.DOCKERHUB_REPO }}",push-by-digest=true,push=true
-
-      - name: Export digest
-        run: |
-          mkdir -p ${{ runner.temp }}/digests
-          digest="${{ steps.build.outputs.digest }}"
-          touch "${{ runner.temp }}/digests/${digest#sha256:}"
-
-      - name: Upload digest
-        uses: actions/upload-artifact@v4
-        with:
-          name: digests-vcpkg-centos-8-${{ matrix.os }}
-          path: ${{ runner.temp }}/digests/*
-          if-no-files-found: error
-          retention-days: 1
+          file: dev/docker/cudf/Dockerfile
+          push: true
+          tags: ${{ env.DOCKERHUB_REPO }}:centos-9-jdk8-cudf
 
-  build-centos-8-jdk8:
+  build-vcpkg-centos-8:
     if: ${{ startsWith(github.repository, 'apache/') }}
     runs-on: ${{ matrix.os }}
     strategy:
@@ -126,7 +101,7 @@ jobs:
         uses: docker/metadata-action@v5
         with:
           images: ${{ env.DOCKERHUB_REPO }}
-          tags: centos-8-jdk8
+          tags: vcpkg-centos-8
 
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
@@ -142,7 +117,7 @@ jobs:
         uses: docker/build-push-action@v6
         with:
           context: .
-          file: dev/docker/Dockerfile.centos8-dynamic-build
+          file: dev/docker/Dockerfile.centos8-static-build
           labels: ${{ steps.meta.outputs.labels }}
           outputs: type=image,"name=${{ env.DOCKERHUB_REPO }}",push-by-digest=true,push=true
 
@@ -155,12 +130,12 @@ jobs:
       - name: Upload digest
         uses: actions/upload-artifact@v4
         with:
-          name: digests-centos-8-jdk8-${{ matrix.os }}
+          name: digests-vcpkg-centos-8-${{ matrix.os }}
           path: ${{ runner.temp }}/digests/*
           if-no-files-found: error
           retention-days: 1
 
-  build-centos-8-jdk11:
+  build-centos-8-jdk8:
     if: ${{ startsWith(github.repository, 'apache/') }}
     runs-on: ${{ matrix.os }}
     strategy:
@@ -176,7 +151,7 @@ jobs:
         uses: docker/metadata-action@v5
         with:
           images: ${{ env.DOCKERHUB_REPO }}
-          tags: centos-8-jdk11
+          tags: centos-8-jdk8
 
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
@@ -192,7 +167,7 @@ jobs:
         uses: docker/build-push-action@v6
         with:
           context: .
-          file: dev/docker/Dockerfile.centos8-dynamic-build-jdk11
+          file: dev/docker/Dockerfile.centos8-dynamic-build
           labels: ${{ steps.meta.outputs.labels }}
           outputs: type=image,"name=${{ env.DOCKERHUB_REPO }}",push-by-digest=true,push=true
 
@@ -205,12 +180,12 @@ jobs:
       - name: Upload digest
         uses: actions/upload-artifact@v4
         with:
-          name: digests-centos-8-jdk11-${{ matrix.os }}
+          name: digests-centos-8-jdk8-${{ matrix.os }}
           path: ${{ runner.temp }}/digests/*
           if-no-files-found: error
           retention-days: 1
 
-  build-centos-8-jdk17:
+  build-centos-8-jdk11:
     if: ${{ startsWith(github.repository, 'apache/') }}
     runs-on: ${{ matrix.os }}
     strategy:
@@ -226,7 +201,7 @@ jobs:
         uses: docker/metadata-action@v5
         with:
           images: ${{ env.DOCKERHUB_REPO }}
-          tags: centos-8-jdk17
+          tags: centos-8-jdk11
 
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
@@ -242,7 +217,7 @@ jobs:
         uses: docker/build-push-action@v6
         with:
           context: .
-          file: dev/docker/Dockerfile.centos8-dynamic-build-jdk17
+          file: dev/docker/Dockerfile.centos8-dynamic-build-jdk11
           labels: ${{ steps.meta.outputs.labels }}
           outputs: type=image,"name=${{ env.DOCKERHUB_REPO }}",push-by-digest=true,push=true
 
@@ -255,12 +230,12 @@ jobs:
       - name: Upload digest
         uses: actions/upload-artifact@v4
         with:
-          name: digests-centos-8-jdk17-${{ matrix.os }}
+          name: digests-centos-8-jdk11-${{ matrix.os }}
           path: ${{ runner.temp }}/digests/*
           if-no-files-found: error
           retention-days: 1
 
-  build-centos-9-jdk8-cudf:
+  build-centos-8-jdk17:
     if: ${{ startsWith(github.repository, 'apache/') }}
     runs-on: ${{ matrix.os }}
     strategy:
@@ -276,7 +251,7 @@ jobs:
         uses: docker/metadata-action@v5
         with:
           images: ${{ env.DOCKERHUB_REPO }}
-          tags: centos-9-jdk8-cudf
+          tags: centos-8-jdk17
 
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
@@ -292,7 +267,7 @@ jobs:
         uses: docker/build-push-action@v6
         with:
           context: .
-          file: dev/docker/cudf/Dockerfile
+          file: dev/docker/Dockerfile.centos8-dynamic-build-jdk17
           labels: ${{ steps.meta.outputs.labels }}
           outputs: type=image,"name=${{ env.DOCKERHUB_REPO }}",push-by-digest=true,push=true
 
@@ -305,7 +280,7 @@ jobs:
       - name: Upload digest
         uses: actions/upload-artifact@v4
         with:
-          name: digests-centos-9-jdk8-cudf-${{ matrix.os }}
+          name: digests-centos-8-jdk17-${{ matrix.os }}
           path: ${{ runner.temp }}/digests/*
           if-no-files-found: error
           retention-days: 1
@@ -315,7 +290,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        digests: [ vcpkg-centos-8, centos-8-jdk8, centos-8-jdk11, centos-8-jdk17, centos-9-jdk8-cudf ]
+        digests: [ vcpkg-centos-8, centos-8-jdk8, centos-8-jdk11, centos-8-jdk17 ]
     needs:
       - build-vcpkg-centos-8
       - build-centos-8-jdk8
diff --git a/dev/docker/cudf/Dockerfile b/dev/docker/cudf/Dockerfile
index e29fc41ad539..6e77b049af5f 100644
--- a/dev/docker/cudf/Dockerfile
+++ b/dev/docker/cudf/Dockerfile
@@ -1,7 +1,7 @@
 FROM ghcr.io/facebookincubator/velox-dev:adapters
 RUN yum install -y sudo patch maven perl
 # After the base docker image updated, don't need to install cmake
-RUN pip install cmake==3.30.4 && ln -s /usr/local/bin/cmake /usr/bin
+RUN pip install cmake==3.30.4 && ln -sf /usr/local/bin/cmake /usr/bin
 
 RUN git clone -b cudf --depth=1 https://github.com/apache/incubator-gluten /opt/gluten
 
diff --git a/ep/build-velox/src/build_velox.sh b/ep/build-velox/src/build_velox.sh
index c79cfc2c1ad2..9239156cedbf 100755
--- a/ep/build-velox/src/build_velox.sh
+++ b/ep/build-velox/src/build_velox.sh
@@ -128,7 +128,8 @@ function compile {
     COMPILE_OPTION="$COMPILE_OPTION -DVELOX_ENABLE_BENCHMARKS=ON"
   fi
   if [ $ENABLE_GPU == "ON" ]; then
-    COMPILE_OPTION="$COMPILE_OPTION -DVELOX_ENABLE_GPU=ON -DVELOX_ENABLE_CUDF=ON"
+    # the cuda default options are for Centos9 image from Meta
+    COMPILE_OPTION="$COMPILE_OPTION -DVELOX_ENABLE_GPU=ON -DVELOX_ENABLE_CUDF=ON -DCMAKE_CUDA_ARCHITECTURES=70 -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.8/bin/nvcc"
   fi
   if [ -n "${GLUTEN_VCPKG_ENABLED:-}" ]; then
     COMPILE_OPTION="$COMPILE_OPTION -DVELOX_GFLAGS_TYPE=static"


Reviews:
user: login: philo-he
body: Looks good!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-15T18:40:08Z
user: login: jinchengchenghh
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T09:10:59Z
user: login: zhouyuan
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-17T21:52:55Z
user: login: zhouyuan
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-18T08:19:00Z
user: login: jinchengchenghh
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T09:42:38Z

Review comments:
diff_hunk: @@ -128,7 +128,8 @@ function compile {
     COMPILE_OPTION="$COMPILE_OPTION -DVELOX_ENABLE_BENCHMARKS=ON"
   fi
   if [ $ENABLE_GPU == "ON" ]; then
-    COMPILE_OPTION="$COMPILE_OPTION -DVELOX_ENABLE_GPU=ON -DVELOX_ENABLE_CUDF=ON"
+    # the cuda default options are for Centos9 image from Meta
+    COMPILE_OPTION="$COMPILE_OPTION -DVELOX_ENABLE_GPU=ON -DVELOX_ENABLE_CUDF=ON -DCMAKE_CUDA_ARCHITECTURES=70 -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.8/bin/nvcc"
path: ep/build-velox/src/build_velox.sh
user: login: jinchengchenghh
body: I use the ENV here but without CUDA_COMPILER https://github.com/apache/incubator-gluten/pull/9333/files#diff-395a263b6f69bd7ef4991face3666e100acdd60e47f4b28e0737adbbf5fb945aR12,is all the environment CUDA COMPILER this path? Do we need to make it an environment variable in docker image?
created_at: 2025-04-17T09:08:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -126,7 +101,7 @@ jobs:
         uses: docker/metadata-action@v5
         with:
           images: ${{ env.DOCKERHUB_REPO }}
-          tags: centos-8-jdk8
+          tags: vcpkg-centos-8
path: .github/workflows/docker_image.yml
user: login: jinchengchenghh
body: Looks like the tag is changed, some docker images in docker hub maybe duplicated. I don't suggest to change the tag, the user may face compatible issue.
created_at: 2025-04-17T09:10:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -126,7 +101,7 @@ jobs:
         uses: docker/metadata-action@v5
         with:
           images: ${{ env.DOCKERHUB_REPO }}
-          tags: centos-8-jdk8
+          tags: vcpkg-centos-8
path: .github/workflows/docker_image.yml
user: login: zhouyuan
body: it's not changed actually - it's diff issue from github. here's the full file:
https://github.com/apache/incubator-gluten/blob/245ea1e22cf81e158fa101aa6b6d970ac17381f9/.github/workflows/docker_image.yml#L63
created_at: 2025-04-17T21:52:55Z
author_association: MEMBER
diff_hunk: @@ -128,7 +128,8 @@ function compile {
     COMPILE_OPTION="$COMPILE_OPTION -DVELOX_ENABLE_BENCHMARKS=ON"
   fi
   if [ $ENABLE_GPU == "ON" ]; then
-    COMPILE_OPTION="$COMPILE_OPTION -DVELOX_ENABLE_GPU=ON -DVELOX_ENABLE_CUDF=ON"
+    # the cuda default options are for Centos9 image from Meta
+    COMPILE_OPTION="$COMPILE_OPTION -DVELOX_ENABLE_GPU=ON -DVELOX_ENABLE_CUDF=ON -DCMAKE_CUDA_ARCHITECTURES=70 -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.8/bin/nvcc"
path: ep/build-velox/src/build_velox.sh
user: login: zhouyuan
body: In my tests it will report `cuda identifier is undefined in device code` if not setting these two variables. Note the GHA runner does not have NV GPU installed.

`CUDA_ARCHITECTURES`is able to read from `CMAKE_CUDA_ARCHITECTURES`
https://cmake.org/cmake/help/v3.28/prop_tgt/CUDA_ARCHITECTURES.html
however it seems there's no similar variable like `CUDA_COMPILERS`

created_at: 2025-04-18T08:19:00Z
author_association: MEMBER

PR comments:
user: login: github-actions[bot]
created_at: 2025-04-15T10:34:30Z
body: https://github.com/apache/incubator-gluten/issues/9243
author_association: NONE
user: login: jinchengchenghh
created_at: 2025-04-17T09:11:18Z
body: Thanks for your fix.
author_association: CONTRIBUTOR
user: login: zhouyuan
created_at: 2025-04-20T10:54:51Z
body: The fix does not work on GHA due to: 
- meta/velox missed one commit to allow skip building tests on GPU enabled 
- GHA runner disk space is not enough on building with meta centos9 adaptors
Will try to make a fix again.
author_association: MEMBER