Repository: containers/ramalama

Issue:
[Feature Request] CPU inferencing with Llama.cpp uses only 4 cores
Running the following command on version v0.6.3 ends up using around 4 cores (4 processes using 100% CPU each).

```bash
ramalama run instructlab/merlinite-7b-lab --ngl 0 "Testing testing testing"
```

Adding `--env LLAMA_ARG_THREADS=32` or something similar does not have an effect.

This is on an AMD HX MAX 395+ (x86) CPU with 16c/32t. But has also been verified on an Ampere platform. So it is very easy to reproduce.

Suspected cause is core autodetection within `llama-run` being broken for some reason and not being able to supply the number of cores manually.

Discussion:
@antheas: cc @KyleGospo @castrojo@hanthor: ```
podman run --rm -i --label ai.ramalama --name ramalama_WUNG8Wm7HY  \
--env=HOME=/tmp --init --security-opt=label=disable --cap-drop=all \
--security-opt=no-new-privileges --label ai.ramalama.model=ollama://llama3.2:3b \
--label ai.ramalama.engine=podman --label ai.ramalama.runtime=llama.cpp \
--label ai.ramalama.port=8080 --label ai.ramalama.command=serve \
--pull=newer -t \
--env LLAMA_ARG_THREADS=18 \
-p 8080:8080 \
--device /dev/dri \
-e INTEL_VISIBLE_DEVICES=1 \
--mount=type=bind,src=/var/home/james/.local/share/ramalama/models/ollama/llama3.2:3b,destination=/mnt/models/model.file,ro quay.io/ramalama/vulkan:latest \
llama-server --port 8080 -m /mnt/models/model.file -c 2048 --temp 0.8 -ngl 0 --host 0.0.0.0 
```

passing LLAMA_ARG_THREADS=18 via the new `--env` works for llama-server but llama-run doesn't appear to support that. I will investigate further

So running `ramalama  --image quay.io/ramalama/vulkan:latest serve --ngl 0 --env=LLAMA_ARG_THREADS=128 ollama://llama3.2:3b` on the Ampere should use all cores @antheas @hanthor: @ericcurtin is there a way to add env var support to llama-run similar to llama-server? thing about the Intel warning

`get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory` also@ericcurtin: > [@ericcurtin](https://github.com/ericcurtin) is there a way to add env var support to llama-run similar to llama-server? thing about the Intel warning
> 
> `get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory` also

Yes since https://github.com/containers/ramalama/issues/934 we can pass any environment variable we want.

I do suspect this is llama.cpp defaulting to 4 threads, we should probably try and default it to number of cores threads I guess.
@antheas: Yeah, also an argument for cores would be nice too. Or making sure the lama cpp variables are parsed to avoid complicating the parser

Defaulting to system cores might break certain workflows that want to share a machine@ericcurtin: Note increasing usage of number of cores may not actually increase performance of the model, I've been doing benchmarks like:

for i in {1..16}; do echo "Threads $i"; build/bin/llama-bench -t $i -m ~/.local/share/ramalama/models/ollama/smollm\:135m; done

I think you can do this already via:

ramalama serve --env LLAMA_ARG_THREADS=12 ...

Wanna try this out @antheas ? And close if necessary?@antheas: ```
❯ sudo ramalama serve instructlab/merlinite-7b-lab --env LLAMA_ARG_THREADS=12
serving on port 8080
Error: configure storage: kernel does not support overlay fs: 'overlay' is not supported over btrfs at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver
```
Not a 1-1 replacement unfortunately.

EDIT: Something broke on my end. I was testing some kernel stuff.@ericcurtin: > ```
> ❯ sudo ramalama serve instructlab/merlinite-7b-lab --env LLAMA_ARG_THREADS=12
> serving on port 8080
> Error: configure storage: kernel does not support overlay fs: 'overlay' is not supported over btrfs at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver
> ```
> 
> Not a 1-1 replacement unfortunately.

Can you describe a 1-1 replacement? A replacement for what exactly?

Something to note, I have an octa-core machine and increasing the number of threads/cores towards 8 utilized just seems to decrease model performance.

> 
> EDIT: Something broke on my end. I was testing some kernel stuff.

@antheas: Sorry, rebooting fixed the error. Being able to put cores on `run` would be nice nonetheless. 

> Something to note, I have an octa-core machine and increasing the number of threads/cores towards 8 utilized just seems to decrease model performance.

CPU? A lot of ultrathin chips struggle with multicore work. Also, it depends on the available ALU units. The Ampere unit has 128 cores.

Here the chip is 16/32 core. Seems like hyperthreading ends up hurting it which happens sometimes but at 1 thread/core we get peak performance. See the benchmark below.

## merlinite-7b-lab

GPU (ROCm):
```
prompt eval time =      69.05 ms /    23 tokens (    3.00 ms per token,   333.11 tokens per second)
       eval time =   15114.94 ms /   513 tokens (   29.46 ms per token,    33.94 tokens per second)
      total time =   15183.99 ms /   536 tokens
```

32 threads:
```
prompt eval time =     302.76 ms /    15 tokens (   20.18 ms per token,    49.54 tokens per second)
       eval time =   82776.73 ms /   952 tokens (   86.95 ms per token,    11.50 tokens per second)
      total time =   83079.49 ms /   967 tokens
```

16 threads:
```
prompt eval time =     363.61 ms /    23 tokens (   15.81 ms per token,    63.25 tokens per second)
       eval time =   88213.42 ms /  2040 tokens (   43.24 ms per token,    23.13 tokens per second)
      total time =   88577.03 ms /  2063 tokens
```

8 threads:
```
prompt eval time =     530.11 ms /    22 tokens (   24.10 ms per token,    41.50 tokens per second)
       eval time =   61218.02 ms /   805 tokens (   76.05 ms per token,    13.15 tokens per second)
      total time =   61748.14 ms /   827 tokens
```

4 threads:
```
prompt eval time =     893.71 ms /    23 tokens (   38.86 ms per token,    25.74 tokens per second)
       eval time =   41978.54 ms /   485 tokens (   86.55 ms per token,    11.55 tokens per second)
      total time =   42872.25 ms /   508 tokens
```

2 threads:
```
prompt eval time =    1627.70 ms /    23 tokens (   70.77 ms per token,    14.13 tokens per second)
       eval time =   76540.61 ms /   691 tokens (  110.77 ms per token,     9.03 tokens per second)
      total time =   78168.31 ms /   714 tokens
```

1 thread:
```
prompt eval time =    3092.34 ms /    23 tokens (  134.45 ms per token,     7.44 tokens per second)
       eval time =  105052.71 ms /   590 tokens (  178.06 ms per token,     5.62 tokens per second)
      total time =  108145.05 ms /   613 tokens
```@ericcurtin: > Sorry, rebooting fixed the error. Being able to put cores on `run` would be nice nonetheless.
> 
> > Something to note, I have an octa-core machine and increasing the number of threads/cores towards 8 utilized just seems to decrease model performance.
> 
> CPU? A lot of ultrathin chips struggle with multicore work. Also, it depends on the available ALU units. The Ampere unit has 128 cores.
> 
> Here the chip is 16/32 core. Seems like hyperthreading ends up hurting it which happens sometimes but at 1 thread/core we get peak performance. See the benchmark below.
> 
> ## merlinite-7b-lab
> GPU (ROCm):
> 
> ```
> prompt eval time =      69.05 ms /    23 tokens (    3.00 ms per token,   333.11 tokens per second)
>        eval time =   15114.94 ms /   513 tokens (   29.46 ms per token,    33.94 tokens per second)
>       total time =   15183.99 ms /   536 tokens
> ```
> 
> 32 threads:
> 
> ```
> prompt eval time =     302.76 ms /    15 tokens (   20.18 ms per token,    49.54 tokens per second)
>        eval time =   82776.73 ms /   952 tokens (   86.95 ms per token,    11.50 tokens per second)
>       total time =   83079.49 ms /   967 tokens
> ```
> 
> 16 threads:
> 
> ```
> prompt eval time =     363.61 ms /    23 tokens (   15.81 ms per token,    63.25 tokens per second)
>        eval time =   88213.42 ms /  2040 tokens (   43.24 ms per token,    23.13 tokens per second)
>       total time =   88577.03 ms /  2063 tokens
> ```
> 
> 8 threads:
> 
> ```
> prompt eval time =     530.11 ms /    22 tokens (   24.10 ms per token,    41.50 tokens per second)
>        eval time =   61218.02 ms /   805 tokens (   76.05 ms per token,    13.15 tokens per second)
>       total time =   61748.14 ms /   827 tokens
> ```
> 
> 4 threads:
> 
> ```
> prompt eval time =     893.71 ms /    23 tokens (   38.86 ms per token,    25.74 tokens per second)
>        eval time =   41978.54 ms /   485 tokens (   86.55 ms per token,    11.55 tokens per second)
>       total time =   42872.25 ms /   508 tokens
> ```
> 
> 2 threads:
> 
> ```
> prompt eval time =    1627.70 ms /    23 tokens (   70.77 ms per token,    14.13 tokens per second)
>        eval time =   76540.61 ms /   691 tokens (  110.77 ms per token,     9.03 tokens per second)
>       total time =   78168.31 ms /   714 tokens
> ```
> 
> 1 thread:
> 
> ```
> prompt eval time =    3092.34 ms /    23 tokens (  134.45 ms per token,     7.44 tokens per second)
>        eval time =  105052.71 ms /   590 tokens (  178.06 ms per token,     5.62 tokens per second)
>       total time =  108145.05 ms /   613 tokens
> ```

Wanna open a PR for the RamaLama-side in python?

I can look at the llama.cpp side of making this change
@antheas: Sure. I can handle the python side. As far as llama.cpp goes it might not be straightforward though. Maybe reusing the existing environment parser somehow should do it @ericcurtin: > Sure. I can handle the python side. As far as llama.cpp goes it might not be straightforward though. Maybe reusing the existing environment parser somehow should do it

I'll handle it, I'm familiar with the llama.cpp code. I'll just copy what llama-server or llama-bench does for llama-run.@hanthor: I did some thread test on an ampere 128 core. On llama 3.2:3b

![Image](https://github.com/user-attachments/assets/c753d84b-acab-4d67-ad4f-a92df032b3b2)
![Image](https://github.com/user-attachments/assets/4e7fa3fb-a368-4df2-8674-2b8f6a9b5813)@ericcurtin: > I did some thread test on an ampere 128 core. On llama 3.2:3b
> 
> ![Image](https://github.com/user-attachments/assets/c753d84b-acab-4d67-ad4f-a92df032b3b2) ![Image](https://github.com/user-attachments/assets/4e7fa3fb-a368-4df2-8674-2b8f6a9b5813)

Cool, seems like we max out at ~ (number of cores)/(2) and after that probably just wasting resources. How were the threads asjusted, was it via:

`llama-bench -t` ?@ericcurtin: @antheas if you are interested in why it maxes out at 4 cores it's here 😄  :

```
ggml/include/ggml.h
228:#define GGML_DEFAULT_N_THREADS  4
```
@ericcurtin: https://github.com/ggml-org/llama.cpp/pull/12370@hanthor: > > I did some thread test on an ampere 128 core. On llama 3.2:3b
> > 
> > ![Image](https://github.com/user-attachments/assets/c753d84b-acab-4d67-ad4f-a92df032b3b2) ![Image](https://github.com/user-attachments/assets/4e7fa3fb-a368-4df2-8674-2b8f6a9b5813)
> 
> Cool, seems like we max out at ~ (number of cores)/(2) and after that probably just wasting resources. How were the threads asjusted, was it via:
> 
> `llama-bench -t` ?

I was doing ramalama serve with the --env LLAMA_MAX_THREADS@hanthor: It would be nice to standardized on -t for bench serve and run. I tested on a 155H and a 13700K and got serious gains increasing threads. I can make more graphs if needed lol@rhatdan: You mean `--threads  10` to set the LLAM_MAX_THREADS environment variable for llama.cpp or something different?@antheas: > [ggml-org/llama.cpp#12370](https://github.com/ggml-org/llama.cpp/pull/12370)

I can drop a PR tomorrow if there's a way to build a container with this PR. I will do `-t` and `--threads`.@hanthor: > You mean `--threads  10` to set the LLAM_MAX_THREADS environment variable for llama.cpp or something different?

Well some standard way to do it between all the commands. Just supporting the envvar is ok, but seeing these performance gains by tweaking this parameter, it argue for something like --threads or -t as the standard way to set it.@antheas: > You mean `--threads 10` to set the LLAM_MAX_THREADS environment variable for llama.cpp or something different?

It seems like llama-run does not support env variables, so for that at least it will be an arg@rhatdan: Don't worry about llama-run, we can either add the support.  In the long run, I want to move to using llama-serve (and vllm) for everything, and then get ramalama run to run a client app that talks to the running service via the API.

This way we can concentrate only on llama-serve and add new features like RAG processing as we go forward.@ericcurtin: I would favour -t FWIW...@rhatdan: I am fine with both -t and --threads as the options. Make sure that you add this to ramalama.conf to allow users to force it on all of their containers, for their specific hardware.

@antheas you got this?@antheas: > I am fine with both -t and --threads as the options. Make sure that you add this to ramalama.conf to allow users to force it on all of their containers, for their specific hardware.
> 
> [@antheas](https://github.com/antheas) you got this?

https://github.com/containers/ramalama/pull/966

> I would favour -t FWIW...

Did both `-t` and `--threads`

Solution patch:
diff --git a/container-images/pragmatic/build_pragmatic.sh b/container-images/pragmatic/build_pragmatic.sh
deleted file mode 100644
index c4ff7f7..0000000
--- a/container-images/pragmatic/build_pragmatic.sh
+++ /dev/null
@@ -1,43 +0,0 @@
-#!/bin/bash
-
-clone_and_build_pragmatic() {
-  git clone https://github.com/redhat-et/PRAGmatic
-  cd PRAGmatic
-  git submodule update --init --recursive
-  local url="https://download.pytorch.org/whl"
-  if [ "$2" == "docling" ]; then
-      ${PYTHON_VERSION} pip install docling --extra-index-url "$url/$1"
-      ${PYTHON_VERSION} pip install -r requirements.txt --prefix=/usr
-  else
-      ${PYTHON_VERSION} pip install torch --extra-index-url "$url/$1"
-      local tmpfile
-      tmpfile=$(mktemp)
-      grep -v docling requirements.txt > "${tmpfile}"
-      ${PYTHON_VERSION} pip install -r "${tmpfile}" --prefix=/usr
-      rm "${tmpfile}"
-  fi
-
-  ${PYTHON_VERSION} pip install --prefix=/usr .
-  cd ..
-}
-
-main() {
-    set -exu -o pipefail
-
-    export PYTHON_VERSION="python3 -m"
-    if [ "$(python3 --version)" \< "Python 3.11" ]; then
-        dnf install -y python3.11 python3.11-pip git
-        export PYTHON_VERSION="/usr/bin/python3.11 -m"
-    else
-        dnf install -y python3-pip git
-    fi
-
-    clone_and_build_pragmatic "$1" "$2"
-    rm -rf /var/cache/*dnf* /opt/rocm-*/lib/*/library/*gfx9* /root/.cache \
-      /root/buildinfo PRAGmatic
-    dnf -y clean all
-    ldconfig
-}
-
-main "$@"
-
diff --git a/container-images/scripts/build_llama_and_whisper.sh b/container-images/scripts/build_llama_and_whisper.sh
index 5ac0707..500c129 100755
--- a/container-images/scripts/build_llama_and_whisper.sh
+++ b/container-images/scripts/build_llama_and_whisper.sh
@@ -166,7 +166,7 @@ clone_and_build_whisper_cpp() {
 }
 
 clone_and_build_llama_cpp() {
-  local llama_cpp_sha="9f2250ba722738ec0e6ab684636268a79160c854"
+  local llama_cpp_sha="8ba95dca2065c0073698afdfcda4c8a8f08bf0d9"
 
   git clone https://github.com/ggml-org/llama.cpp
   cd llama.cpp
@@ -188,6 +188,10 @@ clone_and_build_ramalama() {
   rm -rf ramalama
 }
 
+build_rag() {
+    python3 -m pip install "qdrant-client[fastembed]"  openai
+}
+
 main() {
   # shellcheck disable=SC1091
   source /etc/os-release
@@ -205,6 +209,7 @@ main() {
   available dnf && dnf_install
   if [ -n "$containerfile" ]; then 
       clone_and_build_ramalama
+      build_rag
   fi
   setup_build_env
   clone_and_build_whisper_cpp
diff --git a/container-images/scripts/rag_framework.py b/container-images/scripts/rag_framework.py
new file mode 100644
index 0000000..3a58269
--- /dev/null
+++ b/container-images/scripts/rag_framework.py
@@ -0,0 +1,186 @@
+#!/usr/bin/python3
+
+import hashlib
+import os
+import uuid
+
+import openai
+import qdrant_client
+from docling.chunking import HybridChunker
+from docling.document_converter import DocumentConverter
+
+# Global Vars
+EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
+SPARSE_MODEL = "prithivida/Splade_PP_en_v1"
+COLLECTION_NAME = "rag"
+
+# Helper Classes and Functions
+
+
+class Rag:
+    def __init__(self):
+        self.client = qdrant_client.QdrantClient(":memory:")
+        self.client.set_model(EMBED_MODEL)
+        self.client.set_sparse_model(SPARSE_MODEL)
+        self.conv = Converter()
+        self.llm = openai.OpenAI(api_key="your-api-key", base_url="http://localhost:8080")
+        self.chat_history = []  # Store chat history
+
+    def insertPDF(self, file_path):
+        documents, metadata, ids = self.conv.convert(file_path)
+        self.client.add(COLLECTION_NAME, documents=documents, metadata=metadata, ids=ids)
+
+    def query(self, prompt):
+        # Add user query to chat history
+        self.chat_history.append({"role": "user", "content": prompt})
+
+        # Ensure chat history does not exceed 10 messages (5 user + 5 AI)
+        if len(self.chat_history) > 10:
+            self.chat_history.pop(0)  # Remove the oldest message
+
+        # Query the Qdrant client for relevant context
+        results = self.client.query(
+            collection_name="rag",
+            query_text=prompt,
+            limit=5,
+        )
+        context = "\n".join(r.document for r in results)
+
+        # Prepare the metaprompt with chat history and context
+        metaprompt = f"""
+            You are an expert software architect.
+            Use the provided context and chat history to answer the question accurately and concisely.
+            If the answer is not explicitly stated, infer the most reasonable answer based on the available information.
+            If there is no relevant information, respond with "I don't know"—do not fabricate details.
+
+            ### Chat History:
+            {self.format_chat_history()}
+
+            ### Context:
+            {context.strip()}
+
+            ### Question:
+            {prompt.strip()}
+
+            ### Answer:
+            """
+
+        # Query the LLM with the metaprompt
+        response = self.llm.chat.completions.create(
+            model="your-model-name", messages=[{"role": "user", "content": metaprompt}], stream=True
+        )
+
+        # Collect the AI response and add it to chat history
+        full_response = ""
+        for chunk in response:
+            if chunk.choices[0].delta.content:
+                full_response += chunk.choices[0].delta.content
+                print(chunk.choices[0].delta.content, end="", flush=True)
+
+        # Add AI response to chat history
+        self.chat_history.append({"role": "assistant", "content": full_response})
+
+        # Ensure chat history does not exceed 10 messages after adding the AI response
+        if len(self.chat_history) > 10:
+            self.chat_history.pop(0)  # Remove the oldest message
+
+        print(" ")
+
+    def format_chat_history(self):
+        """Format the chat history into a string for inclusion in the metaprompt."""
+        formatted_history = []
+        for i in range(0, len(self.chat_history), 2):
+            user_message = self.chat_history[i]["content"]
+            if i + 1 < len(self.chat_history):
+                ai_message = self.chat_history[i + 1]["content"]
+                formatted_history.append(f"User: {user_message}\nAI: {ai_message}")
+            else:
+                formatted_history.append(f"User: {user_message}\nAI: ")
+        return "\n".join(formatted_history)
+
+    def run(self):
+        print("> Welcome to the Rag Assistant!")
+        try:
+            while True:
+                # User input
+                user_input = input("> ").strip()
+
+                # Skip empty queries
+                if not user_input:
+                    print("> Please enter a valid query.")
+                    continue
+
+                # Check for a specific query
+                try:
+                    self.query(user_input)
+                except KeyboardInterrupt:
+                    print("\nStream interrupted.")
+                print(" ")
+
+        except KeyboardInterrupt:
+            print("\n> Exiting... Goodbye!")  # Catch any Interrupts and exit gracefully
+
+
+class Converter:
+    """A Class designed to handle all document conversions using Docling"""
+
+    def __init__(self):
+        self.doc_converter = DocumentConverter()
+
+    def convert(self, file_path):
+        targets = []
+
+        # Check if file_path is a directory or a file
+        if os.path.isdir(file_path):
+            targets.extend(self.walk(file_path))  # Walk directory and process all files
+        elif os.path.isfile(file_path):
+            targets.append(file_path)  # Process the single file
+        else:
+            # if the path provided is wrong just return false
+            # Used in Rag.add files to avoid errors
+            return False
+
+        result = self.doc_converter.convert_all(targets)
+
+        documents, metadatas, ids = [], [], []
+
+        chunker = HybridChunker(tokenizer=EMBED_MODEL, max_tokens=500, overlap=100)
+        for file in result:
+            chunk_iter = chunker.chunk(dl_doc=file.document)
+            for i, chunk in enumerate(chunk_iter):
+                doc_text = chunker.serialize(chunk=chunk)
+                # Extract the text and metadata from the chunk
+                doc_meta = chunk.meta.export_json_dict()
+
+                # Append to respective lists
+                documents.append(doc_text)
+                metadatas.append(doc_meta)
+
+                # Generate unique ID for the chunk
+                doc_id = self.generate_hash(doc_text)
+                ids.append(doc_id)
+        return documents, metadatas, ids
+
+    def walk(self, path):
+        targets = []
+        for root, dirs, files in os.walk(path, topdown=True):
+            if len(files) == 0:
+                continue
+            for f in files:
+                file = os.path.join(root, f)
+                if os.path.isfile(file):
+                    targets.append(file)
+        return targets
+
+    def generate_hash(self, document: str) -> str:
+        """Generate a unique hash for a document."""
+        sha256_hash = hashlib.sha256(document.encode('utf-8')).hexdigest()
+
+        # Use the first 32 characters of the hash to create a UUID
+        return str(uuid.UUID(sha256_hash[:32]))
+
+
+rag = Rag()
+file_path = input("Give me file_path of pdf: ")
+rag.insertPDF(file_path)
+rag.run()
diff --git a/container_build.sh b/container_build.sh
index 338317c..5bac45a 100755
--- a/container_build.sh
+++ b/container_build.sh
@@ -27,13 +27,13 @@ add_build_platform() {
   fi
 
   conman_build+=("--platform" "$platform")
-  conman_build+=("-t" "$REGISTRY_PATH/$image_name")
-  conman_build+=("-f" "$image_name/Containerfile" ".")
+  conman_build+=("-t" "$REGISTRY_PATH/${target}")
+  conman_build+=("-f" "${target}/Containerfile" ".")
 }
 
 rm_container_image() {
   if $rm_after_build; then
-    "$conman_bin" rmi -f "$image_name" || true
+    "$conman_bin" rmi -f "${target}" || true
   fi
 }
 
@@ -57,10 +57,10 @@ add_entrypoints() {
 }
 
 build() {
-  cd "container-images"
-  local image_name="${1//container-images\//}"
+  local target=${1}
+  cd "container-images/"
   local conman_build=("${conman[@]}")
-  local conman_show_size=("${conman[@]}" "images" "--filter" "reference='$REGISTRY_PATH/$image_name'")
+  local conman_show_size=("${conman[@]}" "images" "--filter" "reference='$REGISTRY_PATH/${target}'")
   if [ "$3" == "-d" ]; then
       add_build_platform
       echo "${conman_build[@]}"
@@ -74,15 +74,15 @@ build() {
       echo "${conman_build[@]}"
       "${conman_build[@]}"
       "${conman_show_size[@]}"
-      add_entrypoints "${conman[@]}" "$REGISTRY_PATH"/"$image_name"
+      add_entrypoints "${conman[@]}" "${REGISTRY_PATH}"/"${target}"
       rm_container_image
       ;;
     push)
-      "${conman[@]}" push "$REGISTRY_PATH/$image_name"
+      "${conman[@]}" push "$REGISTRY_PATH/${target}"
       ;;
     multi-arch)
-      podman farm build -t "$REGISTRY_PATH"/"$image_name" -f "$image_name"/Containerfile .
-      add_entrypoints "podman farm" "$REGISTRY_PATH"/"$image_name"
+      podman farm build -t "$REGISTRY_PATH"/"${target}" -f "${target}"/Containerfile .
+      add_entrypoints "podman farm" "$REGISTRY_PATH"/"${target}"
       ;;
     *)
       echo "Invalid command: ${2:-}. Use 'build', 'push' or 'multi-arch'."
@@ -150,7 +150,7 @@ process_all_targets() {
   build "container-images/ramalama" "$command" "$option"
   for i in container-images/*; do
     # skip these directories
-    if [[ "$i" =~ ^container-images/(scripts|ramalama|pragmatic)$ ]]; then
+    if [[ "$i" =~ ^container-images/(scripts|ramalama)$ ]]; then
       continue
     fi
 
@@ -218,7 +218,7 @@ main() {
   if [ "$target" = "all" ]; then
     process_all_targets "$command" "$option"
   else
-    build "container-images/$target" "$command" "$option"
+    build "$target" "$command" "$option"
   fi
 }
 
diff --git a/docs/ramalama-bench.1.md b/docs/ramalama-bench.1.md
index 52e83c0..c3584f1 100644
--- a/docs/ramalama-bench.1.md
+++ b/docs/ramalama-bench.1.md
@@ -104,6 +104,10 @@ llama.cpp explains this as:
 
         Usage: Lower numbers are good for virtual assistants where we need deterministic responses. Higher numbers are good for roleplay or creative tasks like editing stories
 
+#### **--threads**, **-t**
+maximum number of cpu threads to use for inferencing
+The default -1, uses the default of the underlying implementation
+
 #### **--tls-verify**=*true*
 require HTTPS and verify certificates when contacting OCI registries
 
diff --git a/docs/ramalama-perplexity.1.md b/docs/ramalama-perplexity.1.md
index 845e33e..a2ec221 100644
--- a/docs/ramalama-perplexity.1.md
+++ b/docs/ramalama-perplexity.1.md
@@ -109,6 +109,10 @@ llama.cpp explains this as:
 
         Usage: Lower numbers are good for virtual assistants where we need deterministic responses. Higher numbers are good for roleplay or creative tasks like editing stories
 
+#### **--threads**, **-t**
+maximum number of cpu threads to use for inferencing
+The default -1, uses the default of the underlying implementation
+
 #### **--tls-verify**=*true*
 require HTTPS and verify certificates when contacting OCI registries
 
diff --git a/docs/ramalama-run.1.md b/docs/ramalama-run.1.md
index 6b24dce..49c95b6 100644
--- a/docs/ramalama-run.1.md
+++ b/docs/ramalama-run.1.md
@@ -65,8 +65,6 @@ set the network mode for the container
 number of gpu layers, 0 means CPU inferencing, 999 means use max layers (default: -1)
 The default -1, means use whatever is automatically deemed appropriate (0 or 999)
 
-Pull image policy. The default is **missing**.
-
 #### **--oci-runtime**
 
 Override the default OCI runtime used to launch the container. Container
@@ -95,6 +93,7 @@ Containers running in a user namespace (e.g., rootless containers) can‐
 not have more privileges than the user that launched them.
 
 #### **--pull**=*policy*
+Pull image policy. The default is **missing**.
 
 - **always**: Always pull the image and throw an error if the pull fails.
 - **missing**: Only pull the image when it does not exist in the local containers storage. Throw an error if no image is found and the pull fails.
@@ -114,6 +113,10 @@ llama.cpp explains this as:
 
     Usage: Lower numbers are good for virtual assistants where we need deterministic responses. Higher numbers are good for roleplay or creative tasks like editing stories
 
+#### **--threads**, **-t**
+maximum number of cpu threads to use for inferencing
+The default -1, uses the default of the underlying implementation
+
 #### **--tls-verify**=*true*
 require HTTPS and verify certificates when contacting OCI registries
 
diff --git a/docs/ramalama-serve.1.md b/docs/ramalama-serve.1.md
index 295fc7d..b1d475b 100644
--- a/docs/ramalama-serve.1.md
+++ b/docs/ramalama-serve.1.md
@@ -143,6 +143,10 @@ llama.cpp explains this as:
 #### **--tls-verify**=*true*
 require HTTPS and verify certificates when contacting OCI registries
 
+#### **--threads**, **-t**
+maximum number of cpu threads to use for inferencing
+The default -1, uses the default of the underlying implementation
+
 ## EXAMPLES
 ### Run two AI Models at the same time. Notice both are running within Podman Containers.
 ```
diff --git a/docs/ramalama.conf b/docs/ramalama.conf
index 22838bd..031ccca 100644
--- a/docs/ramalama.conf
+++ b/docs/ramalama.conf
@@ -99,6 +99,11 @@
 #        Usage: Lower numbers are good for virtual assistants where we need deterministic responses. Higher numbers are good for roleplay or creative tasks like editing stories
 #temp=0.8
 
+# Maximum number of cpu threads to use for inferencing
+# -1 will defer to the underlying implementation
+#
+#threads = -1
+
 # Specify the default transport to be used for pulling and pushing of AI Models.
 # Options: oci, ollama, huggingface.
 #
diff --git a/docs/ramalama.conf.5.md b/docs/ramalama.conf.5.md
index bd58d11..4860bd4 100644
--- a/docs/ramalama.conf.5.md
+++ b/docs/ramalama.conf.5.md
@@ -140,6 +140,11 @@ llama.cpp explains this as:
 
         Usage: Lower numbers are good for virtual assistants where we need deterministic responses. Higher numbers are good for roleplay or creative tasks like editing stories
 
+**threads**=-1
+
+maximum number of cpu threads to use for inferencing
+The default -1, uses the default of the underlying implementation
+
 **transport**="ollama"
 
 Specify the default transport to be used for pulling and pushing of AI Models.
diff --git a/ramalama/cli.py b/ramalama/cli.py
index 3e113c5..fb867ea 100644
--- a/ramalama/cli.py
+++ b/ramalama/cli.py
@@ -713,6 +713,13 @@ def bench_run_serve_perplexity_args(parser):
         default=CONFIG["ngl"],
         help="number of layers to offload to the gpu, if available",
     )
+    parser.add_argument(
+        "-t",
+        "--threads",
+        type=int,
+        default=CONFIG["threads"],
+        help="maximum number of cpu threads to use, if the gpu is unavailable/saturated",
+    )
     parser.add_argument(
         "--oci-runtime",
         help="override the default OCI runtime used to launch the container",
diff --git a/ramalama/common.py b/ramalama/common.py
index c8bf5d4..8ece240 100644
--- a/ramalama/common.py
+++ b/ramalama/common.py
@@ -355,10 +355,10 @@ def check_nvidia():
     try:
         command = ['nvidia-smi']
         run_cmd(command).stdout.decode("utf-8")
-        os.environ["CUDA_VISIBLE_DEVICES"] = "0"
 
         # ensure at least one CDI device resolves
         if resolve_cdi(['/etc/cdi', '/var/run/cdi']):
+            os.environ["CUDA_VISIBLE_DEVICES"] = "0"
             _nvidia = "cuda"
             return _nvidia
 
diff --git a/ramalama/config.py b/ramalama/config.py
index 8476ab2..dc628fc 100644
--- a/ramalama/config.py
+++ b/ramalama/config.py
@@ -74,6 +74,7 @@ def load_config_defaults(config: Dict[str, Any]):
     config['host'] = config.get('host', "0.0.0.0")
     config['keep_groups'] = config.get('keep_groups', False)
     config['ngl'] = config.get('ngl', -1)
+    config['threads'] = config.get('threads', -1)
     config['nocontainer'] = config.get('nocontainer', False)
     config['port'] = config.get('port', int_tuple_as_str(DEFAULT_PORT_RANGE))
     config['pull'] = config.get('pull', "newer")
diff --git a/ramalama/model.py b/ramalama/model.py
index 81f4ba0..4bc257d 100644
--- a/ramalama/model.py
+++ b/ramalama/model.py
@@ -401,6 +401,9 @@ class Model(ModelBase):
 
             gpu_args += [f'{args.ngl}']
 
+        if args.threads != -1:
+            gpu_args += ["--threads", f"{args.threads}"]
+
         return gpu_args
 
     def exec_model_in_container(self, model_path, cmd_args, args):


Test patch:
diff --git a/test/unit/test_config.py b/test/unit/test_config.py
index a50b3fb..cc55ea7 100644
--- a/test/unit/test_config.py
+++ b/test/unit/test_config.py
@@ -67,6 +67,7 @@ def test_load_config_from_env(env, config, expected):
                 },
                 "runtime": "llama.cpp",
                 "ngl": -1,
+                "threads": -1,
                 "keep_groups": False,
                 "ctx_size": 2048,
                 "pull": "newer",
@@ -92,6 +93,7 @@ def test_load_config_from_env(env, config, expected):
                 },
                 "runtime": "llama.cpp",
                 "ngl": -1,
+                "threads": -1,
                 "keep_groups": False,
                 "ctx_size": 2048,
                 "pull": "newer",