Repository: kaito-project/kaito
Pull request #981: test: add more model preset testing config
Author: login: zhuangqh
Labels: name: Review effort 4/5
Created: 2025-04-03T11:03:15Z | Merged: 2025-04-04T00:41:38Z

**Reason for Change**:
<!-- What does this PR improve or fix in Kaito? Why is it needed? -->

**Requirements**

- [x] added unit tests and e2e tests (if applicable).

**Issue Fixed**:
<!-- If this PR fixes GitHub issue 4321, add "Fixes #4321" to the next line. -->

**Notes for Reviewers**:


Diff:
diff --git a/.github/e2e-preset-configs.json b/.github/e2e-preset-configs.json
index bf4537fbb3..9efbd2dacb 100644
--- a/.github/e2e-preset-configs.json
+++ b/.github/e2e-preset-configs.json
@@ -15,7 +15,7 @@
             "gpu_count": 1
           },
           "vllm": {
-            "command": "python3 /workspace/vllm/inference_api.py --dtype float16 --chat-template /workspace/chat_templates/falcon-instruct.jinja",
+            "command": "python3 /workspace/vllm/inference_api.py --kaito-max-probe-steps 6 --dtype float16 --chat-template /workspace/chat_templates/falcon-instruct.jinja",
             "gpu_count": 1
           }
         }
@@ -34,7 +34,7 @@
             "gpu_count": 1
           },
           "vllm": {
-            "command": "python3 /workspace/vllm/inference_api.py --dtype float16 --chat-template /workspace/chat_templates/falcon-instruct.jinja",
+            "command": "python3 /workspace/vllm/inference_api.py --kaito-max-probe-steps 6 --dtype float16 --chat-template /workspace/chat_templates/falcon-instruct.jinja",
             "gpu_count": 1
           }
         }
@@ -91,7 +91,7 @@
             "gpu_count": 1
           },
           "vllm": {
-            "command": "python3 /workspace/vllm/inference_api.py --dtype float16 --chat-template /workspace/chat_templates/mistral-instruct.jinja",
+            "command": "python3 /workspace/vllm/inference_api.py --kaito-max-probe-steps 6 --dtype float16 --chat-template /workspace/chat_templates/mistral-instruct.jinja",
             "gpu_count": 1
           }
         }
@@ -110,7 +110,7 @@
             "gpu_count": 1
           },
           "vllm": {
-            "command": "python3 /workspace/vllm/inference_api.py --dtype float16",
+            "command": "python3 /workspace/vllm/inference_api.py --kaito-max-probe-steps 6 --dtype float16",
             "gpu_count": 1
           }
         }
@@ -129,7 +129,7 @@
             "gpu_count": 1
           },
           "vllm": {
-            "command": "python3 /workspace/vllm/inference_api.py --dtype float16",
+            "command": "python3 /workspace/vllm/inference_api.py --kaito-max-probe-steps 6 --dtype float16",
             "gpu_count": 1
           }
         }
@@ -148,7 +148,7 @@
             "gpu_count": 1
           },
           "vllm": {
-            "command": "python3 /workspace/vllm/inference_api.py --dtype float16",
+            "command": "python3 /workspace/vllm/inference_api.py --kaito-max-probe-steps 6 --dtype float16",
             "gpu_count": 1
           }
         }
@@ -167,7 +167,7 @@
             "gpu_count": 1
           },
           "vllm": {
-            "command": "python3 /workspace/vllm/inference_api.py --dtype float16",
+            "command": "python3 /workspace/vllm/inference_api.py --kaito-config-file /mnt/config/inference_config.yaml --dtype float16",
             "gpu_count": 1
           }
         }
@@ -186,7 +186,7 @@
             "gpu_count": 1
           },
           "vllm": {
-            "command": "python3 /workspace/vllm/inference_api.py --dtype float16 --tensor-parallel-size 2",
+            "command": "python3 /workspace/vllm/inference_api.py --kaito-max-probe-steps 6 --dtype float16 --tensor-parallel-size 2",
             "gpu_count": 2
           }
         }
@@ -205,11 +205,49 @@
             "gpu_count": 1
           },
           "vllm": {
-            "command": "python3 /workspace/vllm/inference_api.py --dtype float16 --max-model-len 1024 --tensor-parallel-size 2",
+            "command": "python3 /workspace/vllm/inference_api.py --kaito-config-file /mnt/config/inference_config.yaml --dtype float16 --max-model-len 1024 --tensor-parallel-size 2",
             "gpu_count": 2
           }
         }
       },
+      {
+        "name": "phi-4",
+        "node-count": 1,
+        "node-vm-size": "Standard_NC24ads_A100_v4",
+        "node-osdisk-size": 100,
+        "OSS": true,
+        "loads_adapter": false,
+        "node_pool": "phi4",
+        "runtimes": {
+          "hf": {
+            "command": "accelerate launch --num_processes 1 --num_machines 1 --machine_rank 0 --gpu_ids all /workspace/tfs/inference_api.py --pipeline text-generation --torch_dtype auto --trust_remote_code",
+            "gpu_count": 1
+          },
+          "vllm": {
+            "command": "python3 /workspace/vllm/inference_api.py --kaito-config-file /mnt/config/inference_config.yaml",
+            "gpu_count": 1
+          }
+        }
+      },
+      {
+        "name": "phi-4-mini-instruct",
+        "node-count": 1,
+        "node-vm-size": "Standard_NC6s_v3",
+        "node-osdisk-size": 100,
+        "OSS": true,
+        "loads_adapter": false,
+        "node_pool": "phi4mini",
+        "runtimes": {
+          "hf": {
+            "command": "accelerate launch --num_processes 1 --num_machines 1 --machine_rank 0 --gpu_ids all /workspace/tfs/inference_api.py --pipeline text-generation --torch_dtype auto --trust_remote_code",
+            "gpu_count": 1
+          },
+          "vllm": {
+            "command": "python3 /workspace/vllm/inference_api.py --kaito-config-file /mnt/config/inference_config.yaml --dtype float16",
+            "gpu_count": 1
+          }
+        }
+      },
       {
         "name": "qwen2.5-coder-7b-instruct",
         "workload": "qwen2-5-coder-7b-instruct",
@@ -230,6 +268,64 @@
           }
         }
       },
+      {
+        "name": "qwen2.5-coder-32b-instruct",
+        "workload": "qwen2-5-coder-32b-instruct",
+        "node-count": 1,
+        "node-vm-size": "Standard_NC24ads_A100_v4",
+        "node-osdisk-size": 200,
+        "OSS": true,
+        "loads_adapter": false,
+        "node_pool": "qwen25c32",
+        "runtimes": {
+          "hf": {
+            "command": "accelerate launch --num_processes 1 --num_machines 1 --machine_rank 0 --gpu_ids all /workspace/tfs/inference_api.py --pipeline text-generation --torch_dtype auto --trust_remote_code",
+            "gpu_count": 1
+          },
+          "vllm": {
+            "command": "python3 /workspace/vllm/inference_api.py --kaito-config-file /mnt/config/inference_config.yaml",
+            "gpu_count": 1
+          }
+        }
+      },
+      {
+        "name": "deepseek-r1-distill-qwen-14b",
+        "node-count": 1,
+        "node-vm-size": "Standard_NC24ads_A100_v4",
+        "node-osdisk-size": 200,
+        "OSS": true,
+        "loads_adapter": false,
+        "node_pool": "dsqwen14b",
+        "runtimes": {
+          "hf": {
+            "command": "accelerate launch --num_processes 1 --num_machines 1 --machine_rank 0 --gpu_ids all /workspace/tfs/inference_api.py --pipeline text-generation --torch_dtype auto --trust_remote_code",
+            "gpu_count": 1
+          },
+          "vllm": {
+            "command": "python3 /workspace/vllm/inference_api.py --kaito-config-file /mnt/config/inference_config.yaml",
+            "gpu_count": 1
+          }
+        }
+      },
+      {
+        "name": "deepseek-r1-distill-llama-8b",
+        "node-count": 1,
+        "node-vm-size": "Standard_NC12s_v3",
+        "node-osdisk-size": 100,
+        "OSS": true,
+        "loads_adapter": false,
+        "node_pool": "dsqwen8b",
+        "runtimes": {
+          "hf": {
+            "command": "accelerate launch --num_processes 1 --num_machines 2 --machine_rank 0 --gpu_ids all /workspace/tfs/inference_api.py --pipeline text-generation --torch_dtype auto --trust_remote_code",
+            "gpu_count": 2
+          },
+          "vllm": {
+            "command": "python3 /workspace/vllm/inference_api.py --kaito-config-file /mnt/config/inference_config.yaml --tensor-parallel-size 2 --max-model-len 32768",
+            "gpu_count": 2
+          }
+        }
+      },
       {
         "name": "llama-2-7b",
         "node-count": 1,
diff --git a/.github/matrix-configs.json b/.github/matrix-configs.json
deleted file mode 100644
index 0b3666742e..0000000000
--- a/.github/matrix-configs.json
+++ /dev/null
@@ -1,100 +0,0 @@
-[
-    {
-        "model": {
-            "runs_on": "self-hosted",
-            "name": "falcon-7b",
-            "dockerfile": "docker/presets/inference/falcon/Dockerfile",
-            "build_args": "--build-arg FALCON_MODEL_NAME=tiiuae/falcon-7b"
-        },
-        "shouldBuildFalcon": "true"
-    },
-    {
-        "model": {
-            "runs_on": "self-hosted",
-            "name": "falcon-7b-instruct",
-            "dockerfile": "docker/presets/inference/falcon/Dockerfile",
-            "build_args": "--build-arg FALCON_MODEL_NAME=tiiuae/falcon-7b-instruct"
-        },
-        "shouldBuildFalcon": "true"
-    },
-
-    {
-        "model": {
-            "runs_on": "self-hosted",
-            "name": "falcon-40b",
-            "dockerfile": "docker/presets/inference/falcon/Dockerfile",
-            "build_args": "--build-arg FALCON_MODEL_NAME=tiiuae/falcon-40b"
-        },
-        "shouldBuildFalcon": "true"
-    },
-
-    {
-        "model": {
-            "runs_on": "self-hosted",
-            "name": "falcon-40b-instruct",
-            "dockerfile": "docker/presets/inference/falcon/Dockerfile",
-            "build_args": "--build-arg FALCON_MODEL_NAME=tiiuae/falcon-40b-instruct"
-        },
-        "shouldBuildFalcon": "true"
-    },
-
-    {
-        "model": {
-            "runs_on": "self-hosted",
-            "name": "llama-2-7b",
-            "dockerfile": "docker/presets/inference/llama-2/Dockerfile",
-            "build_args": "--build-arg LLAMA_WEIGHTS=/llama/llama-2-7b --build-arg SRC_DIR=/home/presets/llama-2"
-        },
-        "shouldBuildLlama2": "true"
-    },
-
-    {
-        "model": {
-            "runs_on": "self-hosted",
-            "name": "llama-2-13b",
-            "dockerfile": "docker/presets/inference/llama-2/Dockerfile",
-            "build_args": "--build-arg LLAMA_WEIGHTS=/llama/llama-2-13b --build-arg SRC_DIR=/home/presets/llama-2"
-        },
-        "shouldBuildLlama2": "true"
-    },
-
-    {
-        "model": {
-            "runs_on": "self-hosted",
-            "name": "llama-2-70b",
-            "dockerfile": "docker/presets/inference/llama-2/Dockerfile",
-            "build_args": "--build-arg LLAMA_WEIGHTS=/llama/llama-2-70b --build-arg SRC_DIR=/home/presets/llama-2"
-        },
-        "shouldBuildLlama2": "true"
-    },
-
-    {
-        "model": {
-            "runs_on": "self-hosted",
-            "name": "llama-2-7b-chat",
-            "dockerfile": "docker/presets/inference/llama-2/Dockerfile",
-            "build_args": "--build-arg LLAMA_WEIGHTS=/llama/llama-2-7b-chat --build-arg SRC_DIR=/home/presets/llama-2-chat"
-        },
-        "shouldBuildLlama2Chat": "true"
-    },
-
-    {
-        "model": {
-            "runs_on": "self-hosted",
-            "name": "llama-2-13b-chat",
-            "dockerfile": "docker/presets/inference/llama-2/Dockerfile",
-            "build_args": "--build-arg LLAMA_WEIGHTS=/llama/llama-2-13b-chat --build-arg SRC_DIR=/home/presets/llama-2-chat"
-        },
-        "shouldBuildLlama2Chat": "true"
-    },
-
-    {
-        "model": {
-            "runs_on": "self-hosted",
-            "name": "llama-2-70b-chat",
-            "dockerfile": "docker/presets/inference/llama-2/Dockerfile",
-            "build_args": "--build-arg LLAMA_WEIGHTS=/llama/llama-2-70b-chat --build-arg SRC_DIR=/home/presets/llama-2-chat"
-        },
-        "shouldBuildLlama2Chat": "true"
-    }
-]
diff --git a/.github/workflows/e2e-base-setup.yaml b/.github/workflows/e2e-base-setup.yaml
deleted file mode 100644
index 93ed1a9b47..0000000000
--- a/.github/workflows/e2e-base-setup.yaml
+++ /dev/null
@@ -1,219 +0,0 @@
-name: e2e-base-setup
-
-on:
-  workflow_call:
-    inputs:
-      git_sha:
-        type: string
-        required: true
-      node_provisioner:
-        type: string
-        required: false
-        default: gpuprovisioner
-      tag:
-        type: string
-      isRelease:
-        type: boolean
-        default: false
-      registry:
-        type: string
-      region:
-        type: string
-        description: "the azure location to run the e2e test in"
-        default: "eastus"
-      k8s_version:
-        type: string
-        default: "1.30.0"
-    outputs:
-      cluster_name:
-        value: ${{ jobs.base-setup.outputs.cluster_name }}
-      registry:
-        value: ${{ jobs.base-setup.outputs.registry }}
-      run_llama_13b:
-        value: ${{ jobs.base-setup.outputs.run_llama_13b }}
-
-jobs:
-  base-setup:
-    runs-on: [ "self-hosted", "hostname:kaito-e2e-github-runner" ]
-    permissions:
-      contents: read
-    environment: e2e-test
-    outputs:
-      cluster_name: ${{ steps.set-outputs.outputs.cluster_name }}
-      registry: ${{ steps.set-outputs.outputs.registry }}
-      run_llama_13b: ${{ steps.set-outputs.outputs.run_llama_13b }}
-
-    env:
-      GO_VERSION: "1.24"
-
-    steps:
-      - name: Harden Runner
-        uses: step-security/harden-runner@4d991eb9b905ef189e4c376166672c3f2f230481 # v2.11.0
-        with:
-          egress-policy: audit
-          disable-sudo: true
-          disable-telemetry: true
-
-      - name: Checkout
-        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
-        with:
-          ref: ${{ inputs.git_sha }}
-
-      - name: Set e2e Resource and Cluster Name
-        run: |
-          ID="ri${{ github.run_id }}rn${{ github.run_number }}ra${{ github.run_attempt }}"
-
-          echo "VERSION=${ID}" >> $GITHUB_ENV
-          echo "CLUSTER_NAME=kaito${ID}" >> $GITHUB_ENV
-          echo "REGISTRY=kaito${ID}.azurecr.io" >> $GITHUB_ENV
-          echo "RUN_LLAMA_13B=false" >> $GITHUB_ENV
-
-      - name: Set Registry
-        if: ${{ inputs.isRelease }}
-        run: |
-          echo "REGISTRY=${{ inputs.registry }}" >> $GITHUB_ENV
-          echo "VERSION=$(echo ${{ inputs.tag }} | tr -d v)" >> $GITHUB_ENV
-
-      - id: set-outputs
-        run: |
-          echo "cluster_name=${CLUSTER_NAME}" >> $GITHUB_OUTPUT
-          echo "registry=${{ env.REGISTRY }}" >> $GITHUB_OUTPUT
-          echo "run_llama_13b=${{ env.RUN_LLAMA_13B }}" >> $GITHUB_OUTPUT
-
-      - name: Remove existing Go modules directory
-        run: sudo rm -rf ~/go/pkg/mod
-
-      - name: Set up Go ${{ env.GO_VERSION }}
-        uses: actions/setup-go@v5.4.0
-        with:
-          go-version: ${{ env.GO_VERSION  }}
-
-      - name: Install Azure CLI latest
-        run: |
-          if ! which az > /dev/null; then
-            echo "Azure CLI not found. Installing..."
-            curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
-          else
-            echo "Azure CLI already installed."
-          fi
-
-      - name: Azure CLI Login
-        run: |
-          az login --identity
-
-      - name: Install Helm
-        uses: azure/setup-helm@v4
-
-      - name: Create Resource Group
-        shell: bash
-        run: |
-          make create-rg
-        env:
-          AZURE_RESOURCE_GROUP: ${{ env.CLUSTER_NAME }}
-
-      - name: Create ACR
-        shell: bash
-        run: |
-          make create-acr
-        env:
-          AZURE_RESOURCE_GROUP: ${{ env.CLUSTER_NAME }}
-          AZURE_ACR_NAME: ${{ env.CLUSTER_NAME }}
-
-      - name: Create Azure Identity
-        uses: azure/CLI@v2.1.0
-        with:
-          inlineScript: |
-            az identity create --name ${{ inputs.node_provisioner }}Identity --resource-group  ${{ env.CLUSTER_NAME }}
-
-      - name: Generate APIs
-        run: |
-          make generate
-
-      - name: build KAITO image
-        if: ${{ !inputs.isRelease }}
-        shell: bash
-        run: |
-          make docker-build-workspace
-        env:
-          REGISTRY: ${{ env.REGISTRY }}
-          VERSION: ${{ env.VERSION }}
-
-      - name: create cluster
-        shell: bash
-        run: |
-          if [ "${{ inputs.node_provisioner }}" == "gpuprovisioner" ]; then
-             make create-aks-cluster
-          else
-            make create-aks-cluster-for-karpenter
-          fi
-        env:
-          AZURE_ACR_NAME: ${{ env.CLUSTER_NAME }}
-          AZURE_RESOURCE_GROUP: ${{ env.CLUSTER_NAME }}
-          AZURE_CLUSTER_NAME: ${{ env.CLUSTER_NAME }}
-          AZURE_LOCATION: ${{ inputs.region }}
-          AKS_K8S_VERSION: ${{ inputs.k8s_version }}
-
-      - name: Create Identities and Permissions for ${{ inputs.node_provisioner }}
-        shell: bash
-        run: |
-          AZURE_SUBSCRIPTION_ID=$E2E_SUBSCRIPTION_ID \
-          make generate-identities
-        env:
-          AZURE_RESOURCE_GROUP: ${{ env.CLUSTER_NAME }}
-          AZURE_CLUSTER_NAME: ${{ env.CLUSTER_NAME }}
-          TEST_SUITE: ${{ inputs.node_provisioner }}
-
-      - name: Install gpu-provisioner helm chart
-        if: ${{ inputs.node_provisioner == 'gpuprovisioner' }}
-        shell: bash
-        run: |
-          AZURE_TENANT_ID=$E2E_TENANT_ID \
-          AZURE_SUBSCRIPTION_ID=$E2E_SUBSCRIPTION_ID \
-          make gpu-provisioner-helm
-        env:
-          AZURE_RESOURCE_GROUP: ${{ env.CLUSTER_NAME }}
-          AZURE_CLUSTER_NAME: ${{ env.CLUSTER_NAME }}
-
-      - name: Install KAITO Workspace helm chart
-        shell: bash
-        run: |
-          make az-patch-install-helm
-          kubectl wait --for=condition=available deploy "kaito-workspace" -n kaito-workspace --timeout=300s
-        env:
-          AZURE_RESOURCE_GROUP: ${{ env.CLUSTER_NAME }}
-          AZURE_CLUSTER_NAME: ${{ env.CLUSTER_NAME }}
-          REGISTRY: ${{ env.REGISTRY }}
-          VERSION: ${{ env.VERSION }}
-          TEST_SUITE: ${{ inputs.node_provisioner }}
-
-      - name: Set up E2E ACR Credentials and Secret
-        shell: bash
-        run: |
-          ACR_USERNAME=$(az acr credential show --name ${{ env.CLUSTER_NAME }} --resource-group ${{ env.CLUSTER_NAME }} --query "username" -o tsv)
-          ACR_PASSWORD=$(az acr credential show --name ${{ env.CLUSTER_NAME }} --resource-group ${{ env.CLUSTER_NAME }} --query "passwords[0].value" -o tsv)
-
-          if [ -z "$ACR_USERNAME" ] || [ -z "$ACR_PASSWORD" ]; then
-            echo "Failed to retrieve ACR credentials"
-            exit 1
-          fi
-
-          kubectl create secret docker-registry ${{ env.CLUSTER_NAME }}-acr-secret \
-            --docker-server=${{ env.CLUSTER_NAME }}.azurecr.io \
-            --docker-username=${ACR_USERNAME} \
-            --docker-password=${ACR_PASSWORD}
-
-      # Add Private-Hosted ACR secret for private models like llama
-      - name: Add Private-Hosted ACR Secret Credentials
-        run: |
-          # Ensure E2E_AMRT_SECRET_NAME is sanitized to remove any accidental quotes
-          E2E_AMRT_SECRET_NAME=$(echo "$E2E_AMRT_SECRET_NAME" | sed 's/[\"'\'']//g')
-
-          if kubectl get secret "$E2E_AMRT_SECRET_NAME" >/dev/null 2>&1; then
-            echo "Secret $E2E_AMRT_SECRET_NAME already exists. Skipping creation."
-          else
-            kubectl create secret docker-registry "$E2E_AMRT_SECRET_NAME" \
-              --docker-server="$E2E_ACR_AMRT_USERNAME.azurecr.io" \
-              --docker-username="$E2E_ACR_AMRT_USERNAME" \
-              --docker-password="$E2E_ACR_AMRT_PASSWORD"
-            echo "Secret $E2E_AMRT_SECRET_NAME created successfully."
-          fi
\ No newline at end of file
diff --git a/.github/workflows/e2e-preset-test.yml b/.github/workflows/e2e-preset-test.yml
index 86ec39e7e9..aea889efb2 100644
--- a/.github/workflows/e2e-preset-test.yml
+++ b/.github/workflows/e2e-preset-test.yml
@@ -176,7 +176,7 @@ jobs:
             echo "WORKLOAD_NAME=$WORKLOAD_NAME" >> $GITHUB_OUTPUT
             WORKLOAD_FILE=$WORKLOAD_NAME.yaml
             echo "WORKLOAD_FILE=$WORKLOAD_FILE" >> $GITHUB_OUTPUT
-            RESOURCE_TYPE=$(echo "${{ matrix.model.name }}" | grep -q "llama" && echo "statefulset" || echo "deployment")
+            RESOURCE_TYPE=$(echo "${{ matrix.model.name }}" | grep -q "llama-2" && echo "statefulset" || echo "deployment")
             echo "RESOURCE_TYPE=$RESOURCE_TYPE" >> $GITHUB_OUTPUT
 
             pip install pyyaml
@@ -225,7 +225,7 @@ jobs:
 
       - name: Wait for Resource to be ready
         run: |
-            kubectl rollout status ${{steps.workload.outputs.RESOURCE_TYPE}}/${{steps.workload.outputs.WORKLOAD_NAME}} --timeout=1800s
+            kubectl rollout status ${{steps.workload.outputs.RESOURCE_TYPE}}/${{steps.workload.outputs.WORKLOAD_NAME}} --timeout=3000s
 
       - name: Check Adapter Loading from Logs
         if: matrix.model.loads_adapter == true
@@ -246,7 +246,7 @@ jobs:
       - name: Test inference endpoint
         run: |
             echo "Testing inference for ${{ matrix.model.name }}"
-            if [[ "${{ matrix.model.name }}" == *"llama"* && "${{ matrix.model.name }}" == *"-chat"* ]]; then
+            if [[ "${{ matrix.model.name }}" == *"llama-2"* && "${{ matrix.model.name }}" == *"-chat"* ]]; then
                 kubectl exec ${{steps.workload.outputs.RESOURCE_TYPE}}/${{steps.workload.outputs.WORKLOAD_NAME}} -- \
                 curl -s -X POST \
                 -H "Content-Type: application/json" \
@@ -267,7 +267,7 @@ jobs:
                     }
                 }' \
                 http://localhost:5000/chat
-            elif [[ "${{ matrix.model.name }}" == *"llama"* ]]; then
+            elif [[ "${{ matrix.model.name }}" == *"llama-2"* ]]; then
                 kubectl exec ${{steps.workload.outputs.RESOURCE_TYPE}}/${{steps.workload.outputs.WORKLOAD_NAME}} -- \
                 curl -s -X POST \
                 -H "Content-Type: application/json" \
diff --git a/.github/workflows/helm-chart.yml b/.github/workflows/helm-chart.yml
index fe9feb0873..0da93e0e55 100644
--- a/.github/workflows/helm-chart.yml
+++ b/.github/workflows/helm-chart.yml
@@ -1,4 +1,4 @@
-name: publish_helm_chart
+name: Publish helm chart
 
 on:
   repository_dispatch:
diff --git a/.github/workflows/lint-go.yml b/.github/workflows/lint-go.yml
index bdaee3f6c0..3a4f1e4865 100644
--- a/.github/workflows/lint-go.yml
+++ b/.github/workflows/lint-go.yml
@@ -1,4 +1,4 @@
-name: lint-go
+name: Go Lint
 
 concurrency:
   group:  ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
diff --git a/.github/workflows/markdown-link-check.yml b/.github/workflows/markdown-link-check.yml
index 37de365db7..a8da377060 100644
--- a/.github/workflows/markdown-link-check.yml
+++ b/.github/workflows/markdown-link-check.yml
@@ -1,4 +1,4 @@
-name: markdown link
+name: Markdown link check
 
 on:
   pull_request:
diff --git a/.github/workflows/pr-title-lint.yml b/.github/workflows/pr-title-lint.yml
index 3dcbf965ad..d4f0714704 100644
--- a/.github/workflows/pr-title-lint.yml
+++ b/.github/workflows/pr-title-lint.yml
@@ -1,4 +1,4 @@
-name: pr Title Checker
+name: PR Title Checker
 on:
   pull_request:
     types:
diff --git a/.github/workflows/publish-workspace-mcr-image.yml b/.github/workflows/publish-workspace-mcr-image.yml
index d24448365f..9ef106d6f1 100644
--- a/.github/workflows/publish-workspace-mcr-image.yml
+++ b/.github/workflows/publish-workspace-mcr-image.yml
@@ -1,4 +1,4 @@
-name: Push image to MCR
+name: Publish KAITO MCR image
 on:
   repository_dispatch:
     types: [ publish-mcr-image ]
diff --git a/.github/workflows/unit-tests.yml b/.github/workflows/unit-tests.yml
index 85b5c0bfec..6b979a31fb 100644
--- a/.github/workflows/unit-tests.yml
+++ b/.github/workflows/unit-tests.yml
@@ -1,4 +1,4 @@
-name: unit-tests
+name: Unit Tests
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
diff --git a/.github/workflows/kaito-e2e.yml b/.github/workflows/workspace-e2e.yml
similarity index 97%
rename from .github/workflows/kaito-e2e.yml
rename to .github/workflows/workspace-e2e.yml
index d0e8af52a5..60176c502c 100644
--- a/.github/workflows/kaito-e2e.yml
+++ b/.github/workflows/workspace-e2e.yml
@@ -1,4 +1,4 @@
-name: pr-e2e-test
+name: E2E Tests
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
diff --git a/Makefile b/Makefile
index 5b8460114a..f21bb413dc 100644
--- a/Makefile
+++ b/Makefile
@@ -48,7 +48,7 @@ AZURE_KARPENTER_MSI_NAME ?= azkarpenterIdentity
 RUN_LLAMA_13B ?= false
 AI_MODELS_REGISTRY ?= modelregistry.azurecr.io
 AI_MODELS_REGISTRY_SECRET ?= modelregistry
-SUPPORTED_MODELS_YAML_PATH ?= ~/runner/_work/kaito/kaito/presets/workspace/models/supported_models.yaml
+SUPPORTED_MODELS_YAML_PATH ?= $(ROOT_DIR)/presets/workspace/models/supported_models.yaml
 
 ## AWS parameters
 CLUSTER_CONFIG_FILE ?= ./docs/aws/clusterconfig.yaml.template
diff --git a/presets/workspace/test/manifests/inference-tmpl/manifest.yaml b/presets/workspace/test/manifests/inference-tmpl/manifest.yaml
index 3609065dab..ad9c34718e 100644
--- a/presets/workspace/test/manifests/inference-tmpl/manifest.yaml
+++ b/presets/workspace/test/manifests/inference-tmpl/manifest.yaml
@@ -4,7 +4,7 @@ deployment:
   metadata:
     name: WORKLOAD_NAME
   spec:
-    progressDeadlineSeconds: 1800
+    progressDeadlineSeconds: 3600
     replicas: 1
     selector:
       matchLabels:
@@ -74,7 +74,7 @@ config:
 
       vllm:
         cpu-offload-gb: 0
-        gpu-memory-utilization: 0.95
+        gpu-memory-utilization: 0.9
         swap-space: 4
         served-model-name: test
         dtype: float16


Reviews:
user: login: zhuangqh
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-03T23:44:45Z
user: login: zhuangqh
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-03T23:47:29Z
user: login: Fei-Guo
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-03T23:47:57Z

Review comments:
diff_hunk: @@ -48,7 +48,7 @@ AZURE_KARPENTER_MSI_NAME ?= azkarpenterIdentity
 RUN_LLAMA_13B ?= false
 AI_MODELS_REGISTRY ?= modelregistry.azurecr.io
 AI_MODELS_REGISTRY_SECRET ?= modelregistry
-SUPPORTED_MODELS_YAML_PATH ?= ~/runner/_work/kaito/kaito/presets/workspace/models/supported_models.yaml
+SUPPORTED_MODELS_YAML_PATH ?= $(ROOT_DIR)/presets/workspace/models/supported_models.yaml
path: Makefile
user: login: zhuangqh
body: We need to pass this env by github env instead.
created_at: 2025-04-03T23:44:45Z
author_association: COLLABORATOR
diff_hunk: @@ -1,100 +0,0 @@
-[
path: .github/matrix-configs.json
user: login: zhuangqh
body: confirmed.It's not needed now.
created_at: 2025-04-03T23:47:11Z
author_association: COLLABORATOR
diff_hunk: @@ -1,219 +0,0 @@
-name: e2e-base-setup
path: .github/workflows/e2e-base-setup.yaml
user: login: zhuangqh
body: confirmed. It's not needed now.
created_at: 2025-04-03T23:47:26Z
author_association: COLLABORATOR

PR comments:
user: login: kaito-pr-agent[bot]
created_at: 2025-04-03T11:03:38Z
body: ## Title

#### (Describe updated until commit https://github.com/kaito-project/kaito/commit/1249176ae1bf22b0c5abb1c5d1580f36fbd52a2e)


test: add more model preset testing config

___
### **User description**
**Reason for Change**:
<!-- What does this PR improve or fix in Kaito? Why is it needed? -->

**Requirements**

- [x] added unit tests and e2e tests (if applicable).

**Issue Fixed**:
<!-- If this PR fixes GitHub issue 4321, add "Fixes #4321" to the next line. -->

**Notes for Reviewers**:


___

### **PR Type**
Enhancement


___

### **Description**
- Added support for new model presets including phi-4, phi-4-mini-instruct, qwen2.5-coder-32b-instruct, deepseek-r1-distill-qwen-14b, and deepseek-r1-distill-llama-8b.

- Updated command arguments for existing model presets to include `--kaito-max-probe-steps` and other configuration options.

- Removed old matrix configurations and workflows related to specific models.

- Adjusted timeout settings for resource readiness checks.


___



### **Changes walkthrough** 📝
<table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>4 files</summary><table>
<tr>
  <td><strong>e2e-preset-configs.json</strong><dd><code>Add support for new model presets</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/981/files#diff-7cdaef2116e347850fb01f1bc2c1c0d90eb240528a5244ddb0bcf591d084eadd">+105/-9</a>&nbsp; </td>

</tr>

<tr>
  <td><strong>e2e-base-setup.yaml</strong><dd><code>Update workflow for new model presets</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/981/files#diff-e23550fa47092beca01283b26f7f2f768e7f31e114388f6b2c74fe66b7c96975">+0/-219</a>&nbsp; </td>

</tr>

<tr>
  <td><strong>e2e-preset-test.yml</strong><dd><code>Update test scripts for new model presets</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/981/files#diff-7c0173625ff2cc5349dfc62ee451a60e49e16acd5a9d10dad7a27b27d616f55f">+4/-4</a>&nbsp; &nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>manifest.yaml</strong><dd><code>Adjust timeout settings</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/981/files#diff-6fb3419f0e77717b65320f3d52389bb96ae10e1bf0c0dd99b7745f085725438b">+2/-2</a>&nbsp; &nbsp; &nbsp; </td>

</tr>
</table></details></td></tr><tr><td><strong>Miscellaneous</strong></td><td><details><summary>2 files</summary><table>
<tr>
  <td><strong>matrix-configs.json</strong><dd><code>Remove old matrix configurations</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/981/files#diff-4577f84f99c9917bc4c63829f33d2c0de3db6025599e2031d56de4c2ba3275e8">+0/-100</a>&nbsp; </td>

</tr>

<tr>
  <td><strong>Makefile</strong><dd><code>Update supported models YAML path</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/981/files#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52">+1/-1</a>&nbsp; &nbsp; &nbsp; </td>

</tr>
</table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>7 files</summary><table>
<tr>
  <td><strong>helm-chart.yml</strong></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/981/files#diff-1d91f5cd0171f7e74ff66564bc76b4ebf62254fcafce4a01f0eb7ff329dd00d1">+1/-1</a>&nbsp; &nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>lint-go.yml</strong></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/981/files#diff-e3eb5891b2343a83d8ea323d0f6ebf2724e3eb90de760155b1926e50f86039e9">+1/-1</a>&nbsp; &nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>markdown-link-check.yml</strong></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/981/files#diff-a778f0a2098e2452bcfe4e72d78a4383bae7e6c6eb8f9f05efc62545640bc7b3">+1/-1</a>&nbsp; &nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>pr-title-lint.yml</strong></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/981/files#diff-00dfab547ea2693d172f71900025e68aad067396af0e3f5d06547b15c464b007">+1/-1</a>&nbsp; &nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>publish-workspace-mcr-image.yml</strong></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/981/files#diff-08e36487096130aae257c684d9bf9f60c29822c5f01f5b5c197e5eea5bb58111">+1/-1</a>&nbsp; &nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>unit-tests.yml</strong></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/981/files#diff-cd0effce06f425599e96952f4d5f684641a683f46d9909e7c74a9f8b2f03ccd3">+1/-1</a>&nbsp; &nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>workspace-e2e.yml</strong></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/981/files#diff-90aa22c0dd263980cdb3167ed3fc7a3a06111bed74ea57632a36f3d9aa15f55b">+1/-1</a>&nbsp; &nbsp; &nbsp; </td>

</tr>
</table></details></td></tr></tr></tbody></table>

___

> <details> <summary>  Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about PR-Agent usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details>
author_association: CONTRIBUTOR
user: login: kaito-pr-agent[bot]
created_at: 2025-04-03T11:03:51Z
body: ## PR Reviewer Guide 🔍

#### (Review updated until commit https://github.com/kaito-project/kaito/commit/1249176ae1bf22b0c5abb1c5d1580f36fbd52a2e)


Here are some key observations to aid the review process:

<table>
<tr><td>⏱️&nbsp;<strong>Estimated effort to review</strong>: 4 🔵🔵🔵🔵⚪</td></tr>
<tr><td>🧪&nbsp;<strong>PR contains tests</strong></td></tr>
<tr><td>🔒&nbsp;<strong>No security concerns identified</strong></td></tr>
<tr><td>⚡&nbsp;<strong>Recommended focus areas for review</strong><br><br>

<details><summary><a href='https://github.com/kaito-project/kaito/pull/981/files#diff-7cdaef2116e347850fb01f1bc2c1c0d90eb240528a5244ddb0bcf591d084eaddR18-R20'><strong>Potential Configuration Error</strong></a>

The `kaito-max-probe-steps` parameter is set to 6 in multiple configurations, but the default value might be sufficient. Verify if this change is necessary and consider adding comments to explain the reasoning behind these values.
</summary>

```json
  "command": "python3 /workspace/vllm/inference_api.py --kaito-max-probe-steps 6 --dtype float16 --chat-template /workspace/chat_templates/falcon-instruct.jinja",
  "gpu_count": 1
}
```

</details>

<details><summary><a href='https://github.com/kaito-project/kaito/pull/981/files#diff-7cdaef2116e347850fb01f1bc2c1c0d90eb240528a5244ddb0bcf591d084eaddR26-R30'><strong>Redundant Configuration</strong></a>

The same configuration is repeated multiple times with slight variations. Consider consolidating these configurations to avoid redundancy and potential inconsistencies.
</summary>

```json
"node-vm-size": "Standard_NC6s_v3",
"node-osdisk-size": 100,
"OSS": true,
"loads_adapter": false,
"node_pool": "falcon7binst",
```

</details>

<details><summary><a href='https://github.com/kaito-project/kaito/pull/981/files#diff-7cdaef2116e347850fb01f1bc2c1c0d90eb240528a5244ddb0bcf591d084eaddR119-R121'><strong>Missing Documentation</strong></a>

Add documentation or comments to explain the purpose and usage of the new configurations, especially those related to new parameters like `kaito-config-file`.
</summary>

```json
"name": "phi-2",
"node-count": 1,
"node-vm-size": "Standard_NC6s_v3",
```

</details>

</td></tr>
</table>

author_association: CONTRIBUTOR
user: login: kaito-pr-agent[bot]
created_at: 2025-04-03T11:04:22Z
body: ## PR Code Suggestions ✨

No code suggestions found for the PR.
author_association: CONTRIBUTOR
user: login: kaito-pr-agent[bot]
created_at: 2025-04-03T11:43:18Z
body: **[Persistent review](https://github.com/kaito-project/kaito/pull/981#issuecomment-2775394290)** updated to latest commit https://github.com/kaito-project/kaito/commit/77411f59c7c8e6ce8f00398f924e880daf1637d6
author_association: CONTRIBUTOR
user: login: kaito-pr-agent[bot]
created_at: 2025-04-03T23:44:55Z
body: **[Persistent review](https://github.com/kaito-project/kaito/pull/981#issuecomment-2775394290)** updated to latest commit https://github.com/kaito-project/kaito/commit/63d19f54981ca0b8ce6d0b0b205c0483f0f98c0f
author_association: CONTRIBUTOR
user: login: zhuangqh
created_at: 2025-04-03T23:57:32Z
body: preset test passed here
vllm  https://github.com/kaito-project/kaito/actions/runs/14242108088/job/39914208127
tfs https://github.com/kaito-project/kaito/actions/runs/14242816022/job/39916501957
 
author_association: COLLABORATOR
user: login: kaito-pr-agent[bot]
created_at: 2025-04-04T00:01:09Z
body: **[Persistent review](https://github.com/kaito-project/kaito/pull/981#issuecomment-2775394290)** updated to latest commit https://github.com/kaito-project/kaito/commit/a287b9ff3d279d6af401cb7230c45db2b839620b
author_association: CONTRIBUTOR
user: login: kaito-pr-agent[bot]
created_at: 2025-04-04T00:40:32Z
body: **[Persistent review](https://github.com/kaito-project/kaito/pull/981#issuecomment-2775394290)** updated to latest commit https://github.com/kaito-project/kaito/commit/1249176ae1bf22b0c5abb1c5d1580f36fbd52a2e
author_association: CONTRIBUTOR