Repository: containers/ramalama
Pull request #1022: Combine Vulkan, Kompute and CPU inferencing into one image
Author: login: ericcurtin
Created: 2025-03-24T11:41:09Z | Merged: 2025-03-27T16:04:24Z

Less images to maintain, Vulkan is more mature and more widely used than Kompute.

## Summary by Sourcery

Enhancements:
- Removes the Kompute specific container image.


Diff:
diff --git a/Makefile b/Makefile
index eedd6d339..ac64a6b88 100644
--- a/Makefile
+++ b/Makefile
@@ -109,6 +109,7 @@ ifneq (,$(wildcard /usr/bin/python3))
 	/usr/bin/python3 -m compileall -q .
 endif
 
+	! grep -ri "#\!/usr/bin/python3" .
 	flake8 *.py */*.py */*/*.py libexec/* bin/*
 	shellcheck *.sh */*.sh */*/*.sh
 
diff --git a/container-images/asahi/Containerfile b/container-images/asahi/Containerfile
index 3dacbf11b..3c54e57e7 100644
--- a/container-images/asahi/Containerfile
+++ b/container-images/asahi/Containerfile
@@ -2,5 +2,6 @@ FROM fedora:41
 
 ENV ASAHI_VISIBLE_DEVICES 1
 COPY --chmod=755 ../scripts /usr/bin
-RUN /usr/bin/build_llama_and_whisper.sh "asahi" && \
+RUN build_llama_and_whisper.sh "asahi" && \
     rag_framework load
+
diff --git a/container-images/cann/Containerfile b/container-images/cann/Containerfile
index 42872fca8..e68a9e6c3 100644
--- a/container-images/cann/Containerfile
+++ b/container-images/cann/Containerfile
@@ -3,8 +3,8 @@ ARG ASCEND_VERSION=cann:8.0.0-910b-openeuler22.03-py3.10
 
 FROM quay.io/ascend/${ASCEND_VERSION} AS builder
 ARG GOLANG_VERSION
-COPY --chmod=755 ../scripts /scripts
-RUN sh -x /scripts/build_llama_and_whisper.sh "cann"
+COPY --chmod=755 ../scripts /usr/bin
+RUN build_llama_and_whisper.sh "cann"
 
 FROM quay.io/ascend/${ASCEND_VERSION}
 # Copy the entire installation directory from the builder
diff --git a/container-images/cuda/Containerfile b/container-images/cuda/Containerfile
index 20a3efec7..6e98884a7 100644
--- a/container-images/cuda/Containerfile
+++ b/container-images/cuda/Containerfile
@@ -1,8 +1,8 @@
 # Base image with CUDA for compilation
 FROM docker.io/nvidia/cuda:12.8.1-devel-ubi9 AS builder
 
-COPY --chmod=755 ../scripts /scripts
-RUN /scripts/build_llama_and_whisper.sh "cuda"
+COPY --chmod=755 ../scripts /usr/bin
+RUN build_llama_and_whisper.sh "cuda"
 
 # Final runtime image
 FROM docker.io/nvidia/cuda:12.8.1-runtime-ubi9
diff --git a/container-images/intel-gpu/Containerfile b/container-images/intel-gpu/Containerfile
index e85edc025..ad4dfa336 100644
--- a/container-images/intel-gpu/Containerfile
+++ b/container-images/intel-gpu/Containerfile
@@ -1,19 +1,18 @@
 FROM quay.io/fedora/fedora:41 as builder
 
 COPY intel-gpu/oneAPI.repo /etc/yum.repos.d/
-COPY --chmod=755 scripts/build_llama_and_whisper.sh /
-
-RUN /build_llama_and_whisper.sh "intel-gpu"
+COPY --chmod=755 ../scripts /usr/bin
+RUN build_llama_and_whisper.sh "intel-gpu"
 
 FROM quay.io/fedora/fedora:41
 
 COPY --from=builder /tmp/install/ /usr/
 COPY intel-gpu/oneAPI.repo /etc/yum.repos.d/
 
-RUN dnf install -y procps-ng python3 python3-pip python3-devel intel-level-zero oneapi-level-zero intel-compute-runtime libcurl lspci clinfo intel-oneapi-runtime-compilers intel-oneapi-mkl-core intel-oneapi-mkl-sycl-blas intel-oneapi-runtime-dnnl ; \
-    chown 0:0 /etc/passwd ; \
-    chown 0:0 /etc/group ; \
-    chmod g=u /etc/passwd /etc/group /home ;
+RUN dnf install -y procps-ng python3 python3-pip python3-devel intel-level-zero oneapi-level-zero intel-compute-runtime libcurl lspci clinfo intel-oneapi-runtime-compilers intel-oneapi-mkl-core intel-oneapi-mkl-sycl-blas intel-oneapi-runtime-dnnl && \
+    chown 0:0 /etc/passwd && \
+    chown 0:0 /etc/group && \
+    chmod g=u /etc/passwd /etc/group /home
 
 RUN useradd llama-user -u 10000 -d /home/llama-user -s /bin/bash && \
 groupmod -a -U llama-user render && \
diff --git a/container-images/ramalama/Containerfile b/container-images/ramalama/Containerfile
index 0080fb0c9..ca103cf3e 100644
--- a/container-images/ramalama/Containerfile
+++ b/container-images/ramalama/Containerfile
@@ -2,5 +2,5 @@ FROM registry.access.redhat.com/ubi9/ubi:9.5-1742918310
 
 COPY --chmod=755 ../scripts /usr/bin
 
-RUN /usr/bin/build_llama_and_whisper.sh "ramalama" && \
+RUN build_llama_and_whisper.sh "ramalama" && \
     rag_framework load
diff --git a/container-images/rocm-fedora/Containerfile b/container-images/rocm-fedora/Containerfile
index 84f531287..9edf4b0ed 100644
--- a/container-images/rocm-fedora/Containerfile
+++ b/container-images/rocm-fedora/Containerfile
@@ -1,7 +1,7 @@
 FROM registry.fedoraproject.org/fedora:42
 
-COPY --chmod=755 ../scripts /usr/bin/
-RUN /usr/bin/build_llama_and_whisper.sh "rocm" && \
+COPY --chmod=755 ../scripts /usr/bin
+RUN build_llama_and_whisper.sh "rocm" && \
     rag_framework load
 
 ENV WHISPER_CPP_SHA=${WHISPER_CPP_SHA}
diff --git a/container-images/rocm/Containerfile b/container-images/rocm/Containerfile
index 97fd23d92..c2b6879ab 100644
--- a/container-images/rocm/Containerfile
+++ b/container-images/rocm/Containerfile
@@ -2,9 +2,9 @@ FROM registry.access.redhat.com/ubi9/ubi:9.5-1742918310
 
 COPY rocm/amdgpu.repo /etc/yum.repos.d/
 COPY rocm/rocm.repo /etc/yum.repos.d/
-COPY --chmod=755 scripts /scripts
+COPY --chmod=755 ../scripts /usr/bin
 
 ARG AMDGPU_TARGETS=
 ENV AMDGPU_TARGETS=${AMDGPU_TARGETS}
-RUN /scripts/build_llama_and_whisper.sh "rocm" && \
+RUN build_llama_and_whisper.sh "rocm" && \
     rag_framework load
diff --git a/container-images/scripts/build_llama_and_whisper.sh b/container-images/scripts/build_llama_and_whisper.sh
index 889509205..c128d9a8c 100755
--- a/container-images/scripts/build_llama_and_whisper.sh
+++ b/container-images/scripts/build_llama_and_whisper.sh
@@ -8,7 +8,7 @@ dnf_install_intel_gpu() {
   local intel_rpms=("intel-oneapi-mkl-sycl-devel" "intel-oneapi-dnnl-devel" \
                   "intel-oneapi-compiler-dpcpp-cpp" "intel-level-zero" \
                   "oneapi-level-zero" "oneapi-level-zero-devel" "intel-compute-runtime")
-  dnf install -y "${rpm_list[@]}" "${intel_rpms[@]}"
+  dnf install -y "${intel_rpms[@]}"
 
   # shellcheck disable=SC1091
   . /opt/intel/oneapi/setvars.sh
@@ -24,11 +24,11 @@ dnf_remove() {
 dnf_install_asahi() {
   dnf copr enable -y @asahi/fedora-remix-branding
   dnf install -y asahi-repos
-  dnf install -y mesa-vulkan-drivers "${vulkan_rpms[@]}" "${rpm_list[@]}"
+  dnf install -y mesa-vulkan-drivers "${vulkan_rpms[@]}"
 }
 
 dnf_install_cuda() {
-  dnf install -y "${rpm_list[@]}" gcc-toolset-12
+  dnf install -y gcc-toolset-12
   # shellcheck disable=SC1091
   . /opt/rh/gcc-toolset-12/enable
 }
@@ -51,9 +51,12 @@ dnf_install_rocm() {
     if [ "${ID}" = "fedora" ]; then
       dnf install -y rocm-core-devel hipblas-devel rocblas-devel rocm-hip-devel
     else
+      add_stream_repo "AppStream"
       dnf install -y rocm-dev hipblas-devel rocblas-devel
     fi
   fi
+
+  rm_non_ubi_repos
 }
 
 dnf_install_s390() {
@@ -73,7 +76,8 @@ add_stream_repo() {
 }
 
 rm_non_ubi_repos() {
-  rm -rf /etc/yum.repos.d/mirror.stream.centos.org_9-stream_* /etc/yum.repos.d/epel*
+  local dir="/etc/yum.repos.d"
+  rm -rf $dir/mirror.stream.centos.org_9-stream_* $dir/epel* $dir/_copr:*
 }
 
 dnf_install_mesa() {
@@ -111,30 +115,27 @@ dnf_install_ffmpeg() {
 }
 
 dnf_install() {
-  local rpm_list=("podman-remote" "python3" "python3-pip" "python3-argcomplete" \
-                  "python3-dnf-plugin-versionlock" "python3-devel" "gcc-c++" "cmake" "vim" \
-                  "procps-ng" "git" "dnf-plugins-core" "libcurl-devel" "gawk")
+  local rpm_list=("podman-remote" "python3" "python3-pip" \
+                  "python3-argcomplete" "python3-dnf-plugin-versionlock" \
+                  "python3-devel" "gcc-c++" "cmake" "vim" "procps-ng" "git" \
+                  "dnf-plugins-core" "libcurl-devel" "gawk")
   local vulkan_rpms=("vulkan-headers" "vulkan-loader-devel" "vulkan-tools" \
                      "spirv-tools" "glslc" "glslang")
-  if [ "${containerfile}" = "ramalama" ] || [[ "${containerfile}" =~ rocm* ]] || \
-    [ "${containerfile}" = "vulkan" ]; then # All the UBI-based ones
-    if [ "${ID}" = "fedora" ]; then
-      dnf install -y "${rpm_list[@]}"
-    else
-      dnf_install_epel
-      dnf --enablerepo=ubi-9-appstream-rpms install -y "${rpm_list[@]}"
-    fi
+  if [ "${ID}" = "fedora" ]; then
+    dnf install -y "${rpm_list[@]}"
+  else
+    dnf_install_epel # All the UBI-based ones
+    dnf --enablerepo=ubi-9-appstream-rpms install -y "${rpm_list[@]}"
+  fi
 
-    # x86_64 and aarch64 means kompute
+  if [ "$containerfile" = "ramalama" ]; then
     if [ "$uname_m" = "x86_64" ] || [ "$uname_m" = "aarch64" ]; then
-      dnf_install_mesa
-    fi
-
-    dnf_install_rocm
-    rm_non_ubi_repos
-    if [ "$uname_m" != "x86_64" ] && ! [ "$uname_m" != "aarch64" ]; then
+      dnf_install_mesa # on x86_64 and aarch64 we use vulkan via mesa 
+    else
       dnf_install_s390
     fi
+  elif [[ "$containerfile" =~ rocm* ]]; then
+    dnf_install_rocm
   elif [ "$containerfile" = "asahi" ]; then
     dnf_install_asahi
   elif [ "$containerfile" = "cuda" ]; then
@@ -203,13 +204,18 @@ configure_common_flags() {
       if [ "${ID}" = "fedora" ]; then
         common_flags+=("-DCMAKE_HIP_COMPILER_ROCM_ROOT=/usr")
       fi
+
       common_flags+=("-DGGML_HIP=ON" "-DAMDGPU_TARGETS=${AMDGPU_TARGETS:-gfx1010,gfx1012,gfx1030,gfx1032,gfx1100,gfx1101,gfx1102,gfx1103,gfx1151,gfx1200,gfx1201}")
       ;;
     cuda)
       common_flags+=("-DGGML_CUDA=ON" "-DCMAKE_EXE_LINKER_FLAGS=-Wl,--allow-shlib-undefined")
       ;;
-    vulkan | asahi)
-      common_flags+=("-DGGML_VULKAN=1")
+    ramalama | asahi)
+      if [ "$uname_m" = "x86_64" ] || [ "$uname_m" = "aarch64" ]; then
+        common_flags+=("-DGGML_VULKAN=ON")
+      else
+        common_flags+=("-DGGML_BLAS=ON" "-DGGML_BLAS_VENDOR=OpenBLAS")
+      fi
       ;;
     intel-gpu)
       common_flags+=("-DGGML_SYCL=ON" "-DCMAKE_C_COMPILER=icx" "-DCMAKE_CXX_COMPILER=icpx")
@@ -259,7 +265,7 @@ clone_and_build_ramalama() {
 }
 
 pip_install() {
-    python3 -m pip install wheel qdrant_client fastembed openai fastapi uvicorn openvino  --prefix="$1"
+  python3 -m pip install wheel qdrant_client fastembed openai fastapi uvicorn openvino  --prefix="$1"
 }
 
 main() {
@@ -278,22 +284,13 @@ main() {
   common_flags+=("-DGGML_CCACHE=OFF" "-DCMAKE_INSTALL_PREFIX=${install_prefix}")
   available dnf && dnf_install
   if [ -n "$containerfile" ]; then 
-      clone_and_build_ramalama "${install_prefix}"
-      pip_install "${install_prefix}"
+    clone_and_build_ramalama "${install_prefix}"
+    pip_install "${install_prefix}"
   fi
+
   setup_build_env
   clone_and_build_whisper_cpp
   common_flags+=("-DLLAMA_CURL=ON")
-  case "$containerfile" in
-    ramalama)
-      if [ "$uname_m" = "x86_64" ] || [ "$uname_m" = "aarch64" ]; then
-        common_flags+=("-DGGML_KOMPUTE=ON" "-DKOMPUTE_OPT_DISABLE_VULKAN_VERSION_CHECK=ON")
-      else
-        common_flags+=("-DGGML_BLAS=ON" "-DGGML_BLAS_VENDOR=OpenBLAS")
-      fi
-      ;;
-  esac
-
   clone_and_build_llama_cpp
   available dnf && dnf_remove
   rm -rf /var/cache/*dnf* /opt/rocm-*/lib/*/library/*gfx9*
diff --git a/container-images/vulkan/Containerfile b/container-images/vulkan/Containerfile
deleted file mode 100644
index eb82b28f5..000000000
--- a/container-images/vulkan/Containerfile
+++ /dev/null
@@ -1,7 +0,0 @@
-FROM registry.access.redhat.com/ubi9/ubi:9.5-1742918310
-
-COPY --chmod=755 ../scripts /usr/bin
-
-RUN /usr/bin/build_llama_and_whisper.sh "vulkan" && \
-    rag_framework load
-


Reviews:
user: login: sourcery-ai[bot]
body: Hey @ericcurtin - I've reviewed your changes - here's some feedback:

**Overall Comments**:
- Consider renaming `ramalama` to `vulkan` to better reflect the change.
- It looks like you're removing the Kompute build, so remove the Kompute flags as well.

<details>
<summary>Here's what I looked at during the review</summary>

- 🟢 **General issues**: all looks good
- 🟢 **Security**: all looks good
- 🟢 **Testing**: all looks good
- 🟢 **Complexity**: all looks good
- 🟢 **Documentation**: all looks good
</details>

***

<details>
<summary>Sourcery is free for open source - if you like our reviews please consider sharing them ✨</summary>

- [X](https://twitter.com/intent/tweet?text=I%20just%20got%20an%20instant%20code%20review%20from%20%40SourceryAI%2C%20and%20it%20was%20brilliant%21%20It%27s%20free%20for%20open%20source%20and%20has%20a%20free%20trial%20for%20private%20code.%20Check%20it%20out%20https%3A//sourcery.ai)
- [Mastodon](https://mastodon.social/share?text=I%20just%20got%20an%20instant%20code%20review%20from%20%40SourceryAI%2C%20and%20it%20was%20brilliant%21%20It%27s%20free%20for%20open%20source%20and%20has%20a%20free%20trial%20for%20private%20code.%20Check%20it%20out%20https%3A//sourcery.ai)
- [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https://sourcery.ai)
- [Facebook](https://www.facebook.com/sharer/sharer.php?u=https://sourcery.ai)

</details>

<sub>
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
</sub>
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T11:42:14Z
user: login: kpouget
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-27T13:52:42Z
user: login: ericcurtin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-27T15:14:59Z

Review comments:
diff_hunk: @@ -2,5 +2,5 @@ FROM registry.access.redhat.com/ubi9/ubi:9.5-1742918310
 
 COPY --chmod=755 ../scripts /usr/bin
 
-RUN /usr/bin/build_llama_and_whisper.sh "ramalama" && \
+RUN build_llama_and_whisper.sh "ramalama"
path: container-images/ramalama/Containerfile
user: login: kpouget
body: missing `&& \` here I think (at least it's inconsistent with the `"rocm"` flavor below)
created_at: 2025-03-27T13:52:42Z
author_association: COLLABORATOR
diff_hunk: @@ -2,5 +2,5 @@ FROM registry.access.redhat.com/ubi9/ubi:9.5-1742918310
 
 COPY --chmod=755 ../scripts /usr/bin
 
-RUN /usr/bin/build_llama_and_whisper.sh "ramalama" && \
+RUN build_llama_and_whisper.sh "ramalama"
path: container-images/ramalama/Containerfile
user: login: ericcurtin
body: Thanks @kpouget nice catch, messed up fixing merge conflicts
created_at: 2025-03-27T15:14:59Z
author_association: MEMBER

PR comments:
user: login: sourcery-ai[bot]
created_at: 2025-03-24T11:41:13Z
body: <!-- Generated by sourcery-ai[bot]: start review_guide -->

## Reviewer's Guide by Sourcery

This pull request consolidates Vulkan, Kompute, and CPU inferencing into a single image. It removes the Kompute-specific configurations and integrates Vulkan support directly into the build script. The separate `vulkan` containerfile was removed.

_No diagrams generated as the changes look simple and do not need a visual representation._

### File-Level Changes

| Change | Details | Files |
| ------ | ------- | ----- |
| Consolidated Vulkan, Kompute, and CPU inferencing into a single image by removing Kompute-specific configurations and integrating Vulkan support directly. | <ul><li>Removed the Kompute-specific conditional compilation flags.</li><li>Enabled Vulkan support for x86_64 and aarch64 architectures.</li><li>For other architectures, enabled OpenBLAS support.</li><li>Removed the separate `vulkan` containerfile.</li></ul> | `container-images/scripts/build_llama_and_whisper.sh`<br/>`container-images/vulkan/Containerfile` |

---

<details>
<summary>Tips and commands</summary>

#### Interacting with Sourcery

- **Trigger a new review:** Comment `@sourcery-ai review` on the pull request.
- **Continue discussions:** Reply directly to Sourcery's review comments.
- **Generate a GitHub issue from a review comment:** Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with `@sourcery-ai issue` to create an issue from it.
- **Generate a pull request title:** Write `@sourcery-ai` anywhere in the pull
  request title to generate a title at any time. You can also comment
  `@sourcery-ai title` on the pull request to (re-)generate the title at any time.
- **Generate a pull request summary:** Write `@sourcery-ai summary` anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment `@sourcery-ai summary` on the pull request to
  (re-)generate the summary at any time.
- **Generate reviewer's guide:** Comment `@sourcery-ai guide` on the pull
  request to (re-)generate the reviewer's guide at any time.
- **Resolve all Sourcery comments:** Comment `@sourcery-ai resolve` on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
- **Dismiss all Sourcery reviews:** Comment `@sourcery-ai dismiss` on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  `@sourcery-ai review` to trigger a new review!
- **Generate a plan of action for an issue:** Comment `@sourcery-ai plan` on
  an issue to generate a plan of action for it.

#### Customizing Your Experience

Access your [dashboard](https://app.sourcery.ai) to:
- Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
- Change the review language.
- Add, remove or edit custom review instructions.
- Adjust other review settings.

#### Getting Help

- [Contact our support team](mailto:support@sourcery.ai) for questions or feedback.
- Visit our [documentation](https://docs.sourcery.ai) for detailed guides and information.
- Keep in touch with the Sourcery team by following us on [X/Twitter](https://x.com/SourceryAI), [LinkedIn](https://www.linkedin.com/company/sourcery-ai/) or [GitHub](https://github.com/sourcery-ai).

</details>

<!-- Generated by sourcery-ai[bot]: end review_guide -->
author_association: CONTRIBUTOR
user: login: ericcurtin
created_at: 2025-03-24T12:21:53Z
body: @bmahabirbu PTAL in the CI some of the Qdrant models seem like they are no longer available on huggingface, would you know what change fixes this?
author_association: MEMBER
user: login: rhatdan
created_at: 2025-03-24T15:27:49Z
body: Hold off on merging this to after the release.  This is good for next week.
author_association: MEMBER
user: login: ericcurtin
created_at: 2025-03-27T13:49:30Z
body: @rhatdan @slp @bmahabirbu PTAL, need to get this into next release for Linaro Connect, wanna demo Vulkan on Apple Silicon
author_association: MEMBER
user: login: ericcurtin
created_at: 2025-03-27T15:34:45Z
body: @kpouget how does it look?
author_association: MEMBER
user: login: kpouget
created_at: 2025-03-27T15:46:56Z
body: > @kpouget how does it look?

looks neat and consistent from what I tell, the PR improves the code quality, and brings in LLama.cpp/Vulkan. Looking forward to perf-testing it once published :) 
author_association: COLLABORATOR
user: login: ericcurtin
created_at: 2025-03-27T16:00:23Z
body: @kpouget perf results will be cool, if you have the hardware around:

metal vs vulkan
rocm vs vulkan
cuda vs vulkan

would all be interesting


author_association: MEMBER
user: login: kpouget
created_at: 2025-03-27T16:08:39Z
body: unfortunately, I only have the hardware for `metal vs vulkan`
I might be able to get my hands on Cuda GPUs for `cuda vs vulkan` testing 🤔 
author_association: COLLABORATOR