Repository: red-hat-data-services/ods-ci
Pull request #2333: model server: move to backend tests in pre/post upgrade
Author: login: rnetser
Labels: name: verified
description: This PR has been tested with Jenkins
name: lgtm
name: approved
Created: 2025-03-30T11:11:13Z | Merged: 2025-04-01T08:56:39Z



Diff:
diff --git a/ods_ci/tests/Resources/Page/ODH/ODHDashboard/ODHModelServing.resource b/ods_ci/tests/Resources/Page/ODH/ODHDashboard/ODHModelServing.resource
index f25de999e..87dc486cc 100644
--- a/ods_ci/tests/Resources/Page/ODH/ODHDashboard/ODHModelServing.resource
+++ b/ods_ci/tests/Resources/Page/ODH/ODHDashboard/ODHModelServing.resource
@@ -395,12 +395,13 @@ Verify Model Inference With Retries
     ...            ${service_port}=${NONE}   ${end_point}=${NONE}   ${set_json_content_type}=${FALSE}
     ${status}=    Run Keyword And Return Status    Verify Model Inference    ${model_name}    ${inference_input}
     ...    ${expected_inference_output}    ${token_auth}    ${project_title}    ${deployment_mode}
+    ...    kserve_mode=${kserve_mode}    service_port=${service_port}   end_point=${end_point}
     ...    set_json_content_type=${set_json_content_type}
     IF    not ${status}
         ${retry}=    Set Variable    ${0}
         WHILE    ${retry} < ${retries}
             IF    ${retry} > 0
-                Log    message=Modelmesh inference call failed ${retry}/${retries}.
+                Log    message=${kserve_mode} inference call failed ${retry}/${retries}.
                 ...    level=WARN
             END
             ${status}=    Run Keyword And Return Status    Verify Model Inference
diff --git a/ods_ci/tests/Tests/0200__rhoai_upgrade/0201__pre_upgrade.robot b/ods_ci/tests/Tests/0200__rhoai_upgrade/0201__pre_upgrade.robot
index 176e4a3e5..94a4bed9d 100644
--- a/ods_ci/tests/Tests/0200__rhoai_upgrade/0201__pre_upgrade.robot
+++ b/ods_ci/tests/Tests/0200__rhoai_upgrade/0201__pre_upgrade.robot
@@ -90,73 +90,46 @@ Verify User Can Disable The Runtime
     [Tags]      Upgrade    ModelServing
     Disable Model Serving Runtime Using CLI     namespace=redhat-ods-applications
 
-Verify Model Can Be Deployed Via UI For Upgrade
+Verify Model Can Be Deployed For Upgrade
     # robocop: off=too-long-test-case
     # robocop: off=too-many-calls-in-test-case
     [Documentation]    Verify Model Can Be Deployed Via UI For Upgrade
-    [Tags]                  Upgrade    ModelServing
-    [Setup]                 Begin Web Test
-    ${PRJ_TITLE}=           Set Variable            model-serving-upgrade
-    ${PRJ_DESCRIPTION}=     Set Variable            project used for model serving tests
-    ${MODEL_NAME}=          Set Variable            test-model
-    ${MODEL_CREATED}=       Set Variable            ${FALSE}
-    ${RUNTIME_NAME}=        Set Variable            Model Serving Test
-    ${INFERENCE_INPUT_OPENVINO}=    Set Variable
-    ...    @tests/Resources/Files/openvino-example-input.json
-    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}=    Set Variable
-    ...    {"model_name":"test-model__isvc-8655dc7979","model_version":"1","outputs":[{"name":"Func/StatefulPartitionedCall/output/_13:0","datatype":"FP32","shape":[1,1],"data":[0.99999994]}]}   # robocop: disable:line-too-long
-    ${runtime_pod_name}=    Replace String Using Regexp
-    ...    string=${RUNTIME_NAME}
-    ...    pattern=\\s
-    ...    replace_with=-
-    ${runtime_pod_name}=    Convert To Lower Case                           ${runtime_pod_name}
-    Fetch CA Certificate If RHODS Is Self-Managed
-    Clean All Models Of Current User
-    Open Data Science Projects Home Page
-    Wait For RHODS Dashboard To Load
-    ...    wait_for_cards=${FALSE}
-    ...    expected_page=Data Science Projects
-    Create Data Science Project     title=${PRJ_TITLE}      description=${PRJ_DESCRIPTION}
-    Create S3 Data Connection
-    ...    project_title=${PRJ_TITLE}
-    ...    dc_name=model-serving-connection
-    ...    aws_access_key=${S3.AWS_ACCESS_KEY_ID}
-    ...    aws_secret_access=${S3.AWS_SECRET_ACCESS_KEY}
-    ...    aws_bucket_name=ods-ci-s3
-    Create Model Server     token=${FALSE}      server_name=${RUNTIME_NAME}
-    Serve Model
-    ...    project_name=${PRJ_TITLE}
-    ...    model_name=${MODEL_NAME}
-    ...    framework=openvino_ir
-    ...    existing_data_connection=${TRUE}
-    ...    data_connection_name=model-serving-connection
-    ...    model_path=openvino-example-model
-    Run Keyword And Continue On Failure
-    ...    Wait Until Keyword Succeeds
-    ...    5 min
-    ...    10 sec
-    ...    Verify Openvino Deployment
-    ...    runtime_name=${runtime_pod_name}
-    ...    project_name=${PRJ_TITLE}
-    Run Keyword And Continue On Failure
-    ...    Wait Until Keyword Succeeds
-    ...    5 min
-    ...    10 sec
-    ...    Verify Serving Service
-    ...    project_name=${PRJ_TITLE}
-    Verify Model Status     ${MODEL_NAME}           success
-    Set Suite Variable      ${MODEL_CREATED}        ${TRUE}     # robocop: disable:replace-set-variable-with-var
-    Run Keyword And Continue On Failure
-    ...    Verify Model Inference
-    ...    ${MODEL_NAME}
-    ...    ${INFERENCE_INPUT_OPENVINO}
-    ...    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}
-    ...    token_auth=${FALSE}
-    Remove File     openshift_ca.crt
-    [Teardown]      Run Keywords        Dashboard Test Teardown
-    ...     AND
-    ...     Run Keyword If Test Failed      Get Events And Pod Logs     namespace=${PRJ_TITLE}
-    ...     label_selector=name=modelmesh-serving-${runtime_pod_name}
+    [Tags]                  Upgrade    ModelServing    ModelServer
+    Set Suite Variable    ${TEST_NS}    ovmsmodel-upgrade
+    Set Suite Variable    ${KSERVE_MODE}    RawDeployment    # RawDeployment   # Serverless
+    Set Suite Variable    ${MODELS_BUCKET}    ${S3.BUCKET_1}
+    Set Suite Variable    ${INFERENCE_INPUT}    @tests/Resources/Files/modelmesh-mnist-input.json
+    Set Suite Variable    ${EXPECTED_INFERENCE_OUTPUT}    {"model_name": "test-dir","model_version": "1","outputs": [{"name": "Plus214_Output_0","shape": [1, 10],"datatype": "FP32","data": [-8.233053207397461, -7.749703407287598, -3.4236814975738527, 12.363029479980469, -12.079103469848633, 17.2665958404541, -10.570976257324219, 0.7130761742591858, 3.3217151165008547, 1.3621227741241456]}]}  #robocop: disable
+    Setup Test Variables    model_name=test-dir    use_pvc=${TRUE}    use_gpu=${FALSE}
+    ...    kserve_mode=${KSERVE_MODE}
+    Set Project And Runtime    runtime=ovms-runtime     protocol=http     namespace=${TEST_NS}
+    ...    download_in_pvc=${TRUE}    model_name=${model_name}
+    ...    storage_size=100Mi    memory_request=100Mi
+    ${requests}=    Create Dictionary    memory=1Gi
+    Compile Inference Service YAML    isvc_name=${model_name}
+    ...    sa_name=${EMPTY}
+    ...    model_storage_uri=pvc://${model_name}-claim/${model_name}
+    ...    model_format=onnx    serving_runtime=ovms-runtime
+    ...    limits_dict=&{EMPTY}    requests_dict=${requests}    kserve_mode=${KSERVE_MODE}
+    Deploy Model Via CLI    isvc_filepath=${INFERENCESERVICE_FILLED_FILEPATH}
+    ...    namespace=${TEST_NS}
+    # File is not needed anymore after applying
+    Remove File    ${INFERENCESERVICE_FILLED_FILEPATH}
+    Wait For Pods To Be Ready    label_selector=serving.kserve.io/inferenceservice=${model_name}
+    ...    namespace=${TEST_NS}
+    ${pod_name}=  Get Pod Name    namespace=${TEST_NS}
+    ...    label_selector=serving.kserve.io/inferenceservice=${model_name}
+    ${service_port}=    Extract Service Port    service_name=${model_name}-predictor    protocol=TCP
+    ...    namespace=${TEST_NS}
+    IF   "${KSERVE_MODE}"=="RawDeployment"
+        Start Port-forwarding    namespace=${TEST_NS}    pod_name=${pod_name}  local_port=${service_port}
+        ...    remote_port=${service_port}    process_alias=ovms-process
+    END
+    Verify Model Inference With Retries   model_name=${model_name}    inference_input=${INFERENCE_INPUT}
+    ...    expected_inference_output=${EXPECTED_INFERENCE_OUTPUT}   project_title=${TEST_NS}
+    ...    deployment_mode=Cli  kserve_mode=${KSERVE_MODE}    service_port=${service_port}
+    ...    end_point=/v2/models/${model_name}/infer  retries=10
+
 
 Verify User Can Deploy Custom Runtime For Upgrade
     [Documentation]     Verify User Can Deploy Custom Runtime For Upgrade
diff --git a/ods_ci/tests/Tests/0200__rhoai_upgrade/0203__post_upgrade.robot b/ods_ci/tests/Tests/0200__rhoai_upgrade/0203__post_upgrade.robot
index c7ea783c0..e70b783a5 100644
--- a/ods_ci/tests/Tests/0200__rhoai_upgrade/0203__post_upgrade.robot
+++ b/ods_ci/tests/Tests/0200__rhoai_upgrade/0203__post_upgrade.robot
@@ -156,30 +156,29 @@ Verify POD Status
 Test Inference Post RHODS Upgrade
     # robocop: off=too-many-calls-in-test-case
     # robocop: off=too-long-test-case
-    [Documentation]    Test the inference result after having deployed a model that requires Token Authentication
-    [Tags]                  Upgrade    ModelServing
-    [Setup]                 Begin Web Test
-    ${PRJ_TITLE}            Set Variable        model-serving-upgrade
-    ${PRJ_DESCRIPTION}      Set Variable        project used for model serving tests        # robocop: off=unused-variable      # robocop: disable:line-too-long
-    ${MODEL_NAME}           Set Variable        test-model
-    ${MODEL_CREATED}        Set Variable        ${FALSE}        # robocop: off=unused-variable
-    ${RUNTIME_NAME}         Set Variable        Model Serving Test      # robocop: off=unused-variable
-    ${INFERENCE_INPUT}      Set Variable        @tests/Resources/Files/modelmesh-mnist-input.json       # robocop: off=unused-variable      # robocop: disable:line-too-long
-    ${INFERENCE_INPUT_OPENVINO}    Set Variable
-    ...    @tests/Resources/Files/openvino-example-input.json
-    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}    Set Variable
-    ...    {"model_name":"test-model__isvc-8655dc7979","model_version":"1","outputs":[{"name":"Func/StatefulPartitionedCall/output/_13:0","datatype":"FP32","shape":[1,1],"data":[0.99999994]}]}   # robocop: disable:line-too-long
-    Fetch CA Certificate If RHODS Is Self-Managed
-    Open Model Serving Home Page
-    Verify Model Status     ${MODEL_NAME}           success
-    Run Keyword And Continue On Failure
-    ...    Verify Model Inference
-    ...    ${MODEL_NAME}
-    ...    ${INFERENCE_INPUT_OPENVINO}
-    ...    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}
-    ...    token_auth=${FALSE}
-    Remove File     openshift_ca.crt
-    [Teardown]      Run     oc delete project ${PRJ_TITLE}
+    [Documentation]    Test the inference result after having deployed a model
+    [Tags]                  Upgrade    ModelServing    ModelServer
+    Set Suite Variable    ${TEST_NS}    ovmsmodel-upgrade
+    Set Suite Variable    ${KSERVE_MODE}    RawDeployment    # RawDeployment   # Serverless
+    Set Suite Variable    ${INFERENCE_INPUT}    @tests/Resources/Files/modelmesh-mnist-input.json
+    Set Suite Variable    ${EXPECTED_INFERENCE_OUTPUT}    {"model_name": "test-dir","model_version": "1","outputs": [{"name": "Plus214_Output_0","shape": [1, 10],"datatype": "FP32","data": [-8.233053207397461, -7.749703407287598, -3.4236814975738527, 12.363029479980469, -12.079103469848633, 17.2665958404541, -10.570976257324219, 0.7130761742591858, 3.3217151165008547, 1.3621227741241456]}]}  #robocop: disable
+    Setup Test Variables    model_name=test-dir    use_pvc=${TRUE}    use_gpu=${FALSE}
+    ...    kserve_mode=${KSERVE_MODE}
+    Wait For Pods To Be Ready    label_selector=serving.kserve.io/inferenceservice=${model_name}
+    ...    namespace=${TEST_NS}
+    ${pod_name}=  Get Pod Name    namespace=${TEST_NS}
+    ...    label_selector=serving.kserve.io/inferenceservice=${model_name}
+    ${service_port}=    Extract Service Port    service_name=${model_name}-predictor    protocol=TCP
+    ...    namespace=${TEST_NS}
+    IF   "${KSERVE_MODE}"=="RawDeployment"
+        Start Port-forwarding    namespace=${TEST_NS}    pod_name=${pod_name}  local_port=${service_port}
+        ...    remote_port=${service_port}    process_alias=ovms-process
+    END
+    Verify Model Inference With Retries   model_name=${model_name}    inference_input=${INFERENCE_INPUT}
+    ...    expected_inference_output=${EXPECTED_INFERENCE_OUTPUT}   project_title=${TEST_NS}
+    ...    deployment_mode=Cli  kserve_mode=${KSERVE_MODE}    service_port=${service_port}
+    ...    end_point=/v2/models/${model_name}/infer  retries=10
+    [Teardown]      Run     oc delete project ${TEST_NS}
 
 Verify Custom Runtime Exists After Upgrade
     [Documentation]    Test the inference result after having deployed a model that requires Token Authentication


Reviews:
user: login: kobihk
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-30T12:38:07Z
user: login: github-advanced-security[bot]
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-01T08:46:50Z
user: login: mwaykole
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-01T08:46:57Z

Review comments:
diff_hunk: @@ -90,73 +90,46 @@
     [Tags]      Upgrade    ModelServing
     Disable Model Serving Runtime Using CLI     namespace=redhat-ods-applications
 
-Verify Model Can Be Deployed Via UI For Upgrade
+Verify Model Can Be Deployed For Upgrade
     # robocop: off=too-long-test-case
     # robocop: off=too-many-calls-in-test-case
     [Documentation]    Verify Model Can Be Deployed Via UI For Upgrade
-    [Tags]                  Upgrade    ModelServing
-    [Setup]                 Begin Web Test
-    ${PRJ_TITLE}=           Set Variable            model-serving-upgrade
-    ${PRJ_DESCRIPTION}=     Set Variable            project used for model serving tests
-    ${MODEL_NAME}=          Set Variable            test-model
-    ${MODEL_CREATED}=       Set Variable            ${FALSE}
-    ${RUNTIME_NAME}=        Set Variable            Model Serving Test
-    ${INFERENCE_INPUT_OPENVINO}=    Set Variable
-    ...    @tests/Resources/Files/openvino-example-input.json
-    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}=    Set Variable
-    ...    {"model_name":"test-model__isvc-8655dc7979","model_version":"1","outputs":[{"name":"Func/StatefulPartitionedCall/output/_13:0","datatype":"FP32","shape":[1,1],"data":[0.99999994]}]}   # robocop: disable:line-too-long
-    ${runtime_pod_name}=    Replace String Using Regexp
-    ...    string=${RUNTIME_NAME}
-    ...    pattern=\\s
-    ...    replace_with=-
-    ${runtime_pod_name}=    Convert To Lower Case                           ${runtime_pod_name}
-    Fetch CA Certificate If RHODS Is Self-Managed
-    Clean All Models Of Current User
-    Open Data Science Projects Home Page
-    Wait For RHODS Dashboard To Load
-    ...    wait_for_cards=${FALSE}
-    ...    expected_page=Data Science Projects
-    Create Data Science Project     title=${PRJ_TITLE}      description=${PRJ_DESCRIPTION}
-    Create S3 Data Connection
-    ...    project_title=${PRJ_TITLE}
-    ...    dc_name=model-serving-connection
-    ...    aws_access_key=${S3.AWS_ACCESS_KEY_ID}
-    ...    aws_secret_access=${S3.AWS_SECRET_ACCESS_KEY}
-    ...    aws_bucket_name=ods-ci-s3
-    Create Model Server     token=${FALSE}      server_name=${RUNTIME_NAME}
-    Serve Model
-    ...    project_name=${PRJ_TITLE}
-    ...    model_name=${MODEL_NAME}
-    ...    framework=openvino_ir
-    ...    existing_data_connection=${TRUE}
-    ...    data_connection_name=model-serving-connection
-    ...    model_path=openvino-example-model
-    Run Keyword And Continue On Failure
-    ...    Wait Until Keyword Succeeds
-    ...    5 min
-    ...    10 sec
-    ...    Verify Openvino Deployment
-    ...    runtime_name=${runtime_pod_name}
-    ...    project_name=${PRJ_TITLE}
-    Run Keyword And Continue On Failure
-    ...    Wait Until Keyword Succeeds
-    ...    5 min
-    ...    10 sec
-    ...    Verify Serving Service
-    ...    project_name=${PRJ_TITLE}
-    Verify Model Status     ${MODEL_NAME}           success
-    Set Suite Variable      ${MODEL_CREATED}        ${TRUE}     # robocop: disable:replace-set-variable-with-var
-    Run Keyword And Continue On Failure
-    ...    Verify Model Inference
-    ...    ${MODEL_NAME}
-    ...    ${INFERENCE_INPUT_OPENVINO}
-    ...    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}
-    ...    token_auth=${FALSE}
-    Remove File     openshift_ca.crt
-    [Teardown]      Run Keywords        Dashboard Test Teardown
-    ...     AND
-    ...     Run Keyword If Test Failed      Get Events And Pod Logs     namespace=${PRJ_TITLE}
-    ...     label_selector=name=modelmesh-serving-${runtime_pod_name}
+    [Tags]                  Upgrade    ModelServing    ModelServer
+    Set Suite Variable    ${TEST_NS}    ovmsmodel-upgrade
path: ods_ci/tests/Tests/0200__rhoai_upgrade/0201__pre_upgrade.robot
user: login: github-advanced-security[bot]
body: ## {{ set_variable_keyword }} can be replaced with VAR

Set Suite Variable can be replaced with VAR

[Show more details](https://github.com/red-hat-data-services/ods-ci/security/code-scanning/28742)
created_at: 2025-04-01T08:46:49Z
author_association: NONE
diff_hunk: @@ -90,73 +90,46 @@
     [Tags]      Upgrade    ModelServing
     Disable Model Serving Runtime Using CLI     namespace=redhat-ods-applications
 
-Verify Model Can Be Deployed Via UI For Upgrade
+Verify Model Can Be Deployed For Upgrade
     # robocop: off=too-long-test-case
     # robocop: off=too-many-calls-in-test-case
     [Documentation]    Verify Model Can Be Deployed Via UI For Upgrade
-    [Tags]                  Upgrade    ModelServing
-    [Setup]                 Begin Web Test
-    ${PRJ_TITLE}=           Set Variable            model-serving-upgrade
-    ${PRJ_DESCRIPTION}=     Set Variable            project used for model serving tests
-    ${MODEL_NAME}=          Set Variable            test-model
-    ${MODEL_CREATED}=       Set Variable            ${FALSE}
-    ${RUNTIME_NAME}=        Set Variable            Model Serving Test
-    ${INFERENCE_INPUT_OPENVINO}=    Set Variable
-    ...    @tests/Resources/Files/openvino-example-input.json
-    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}=    Set Variable
-    ...    {"model_name":"test-model__isvc-8655dc7979","model_version":"1","outputs":[{"name":"Func/StatefulPartitionedCall/output/_13:0","datatype":"FP32","shape":[1,1],"data":[0.99999994]}]}   # robocop: disable:line-too-long
-    ${runtime_pod_name}=    Replace String Using Regexp
-    ...    string=${RUNTIME_NAME}
-    ...    pattern=\\s
-    ...    replace_with=-
-    ${runtime_pod_name}=    Convert To Lower Case                           ${runtime_pod_name}
-    Fetch CA Certificate If RHODS Is Self-Managed
-    Clean All Models Of Current User
-    Open Data Science Projects Home Page
-    Wait For RHODS Dashboard To Load
-    ...    wait_for_cards=${FALSE}
-    ...    expected_page=Data Science Projects
-    Create Data Science Project     title=${PRJ_TITLE}      description=${PRJ_DESCRIPTION}
-    Create S3 Data Connection
-    ...    project_title=${PRJ_TITLE}
-    ...    dc_name=model-serving-connection
-    ...    aws_access_key=${S3.AWS_ACCESS_KEY_ID}
-    ...    aws_secret_access=${S3.AWS_SECRET_ACCESS_KEY}
-    ...    aws_bucket_name=ods-ci-s3
-    Create Model Server     token=${FALSE}      server_name=${RUNTIME_NAME}
-    Serve Model
-    ...    project_name=${PRJ_TITLE}
-    ...    model_name=${MODEL_NAME}
-    ...    framework=openvino_ir
-    ...    existing_data_connection=${TRUE}
-    ...    data_connection_name=model-serving-connection
-    ...    model_path=openvino-example-model
-    Run Keyword And Continue On Failure
-    ...    Wait Until Keyword Succeeds
-    ...    5 min
-    ...    10 sec
-    ...    Verify Openvino Deployment
-    ...    runtime_name=${runtime_pod_name}
-    ...    project_name=${PRJ_TITLE}
-    Run Keyword And Continue On Failure
-    ...    Wait Until Keyword Succeeds
-    ...    5 min
-    ...    10 sec
-    ...    Verify Serving Service
-    ...    project_name=${PRJ_TITLE}
-    Verify Model Status     ${MODEL_NAME}           success
-    Set Suite Variable      ${MODEL_CREATED}        ${TRUE}     # robocop: disable:replace-set-variable-with-var
-    Run Keyword And Continue On Failure
-    ...    Verify Model Inference
-    ...    ${MODEL_NAME}
-    ...    ${INFERENCE_INPUT_OPENVINO}
-    ...    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}
-    ...    token_auth=${FALSE}
-    Remove File     openshift_ca.crt
-    [Teardown]      Run Keywords        Dashboard Test Teardown
-    ...     AND
-    ...     Run Keyword If Test Failed      Get Events And Pod Logs     namespace=${PRJ_TITLE}
-    ...     label_selector=name=modelmesh-serving-${runtime_pod_name}
+    [Tags]                  Upgrade    ModelServing    ModelServer
+    Set Suite Variable    ${TEST_NS}    ovmsmodel-upgrade
+    Set Suite Variable    ${KSERVE_MODE}    RawDeployment    # RawDeployment   # Serverless
path: ods_ci/tests/Tests/0200__rhoai_upgrade/0201__pre_upgrade.robot
user: login: github-advanced-security[bot]
body: ## {{ set_variable_keyword }} can be replaced with VAR

Set Suite Variable can be replaced with VAR

[Show more details](https://github.com/red-hat-data-services/ods-ci/security/code-scanning/28743)
created_at: 2025-04-01T08:46:49Z
author_association: NONE
diff_hunk: @@ -90,73 +90,46 @@
     [Tags]      Upgrade    ModelServing
     Disable Model Serving Runtime Using CLI     namespace=redhat-ods-applications
 
-Verify Model Can Be Deployed Via UI For Upgrade
+Verify Model Can Be Deployed For Upgrade
     # robocop: off=too-long-test-case
     # robocop: off=too-many-calls-in-test-case
     [Documentation]    Verify Model Can Be Deployed Via UI For Upgrade
-    [Tags]                  Upgrade    ModelServing
-    [Setup]                 Begin Web Test
-    ${PRJ_TITLE}=           Set Variable            model-serving-upgrade
-    ${PRJ_DESCRIPTION}=     Set Variable            project used for model serving tests
-    ${MODEL_NAME}=          Set Variable            test-model
-    ${MODEL_CREATED}=       Set Variable            ${FALSE}
-    ${RUNTIME_NAME}=        Set Variable            Model Serving Test
-    ${INFERENCE_INPUT_OPENVINO}=    Set Variable
-    ...    @tests/Resources/Files/openvino-example-input.json
-    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}=    Set Variable
-    ...    {"model_name":"test-model__isvc-8655dc7979","model_version":"1","outputs":[{"name":"Func/StatefulPartitionedCall/output/_13:0","datatype":"FP32","shape":[1,1],"data":[0.99999994]}]}   # robocop: disable:line-too-long
-    ${runtime_pod_name}=    Replace String Using Regexp
-    ...    string=${RUNTIME_NAME}
-    ...    pattern=\\s
-    ...    replace_with=-
-    ${runtime_pod_name}=    Convert To Lower Case                           ${runtime_pod_name}
-    Fetch CA Certificate If RHODS Is Self-Managed
-    Clean All Models Of Current User
-    Open Data Science Projects Home Page
-    Wait For RHODS Dashboard To Load
-    ...    wait_for_cards=${FALSE}
-    ...    expected_page=Data Science Projects
-    Create Data Science Project     title=${PRJ_TITLE}      description=${PRJ_DESCRIPTION}
-    Create S3 Data Connection
-    ...    project_title=${PRJ_TITLE}
-    ...    dc_name=model-serving-connection
-    ...    aws_access_key=${S3.AWS_ACCESS_KEY_ID}
-    ...    aws_secret_access=${S3.AWS_SECRET_ACCESS_KEY}
-    ...    aws_bucket_name=ods-ci-s3
-    Create Model Server     token=${FALSE}      server_name=${RUNTIME_NAME}
-    Serve Model
-    ...    project_name=${PRJ_TITLE}
-    ...    model_name=${MODEL_NAME}
-    ...    framework=openvino_ir
-    ...    existing_data_connection=${TRUE}
-    ...    data_connection_name=model-serving-connection
-    ...    model_path=openvino-example-model
-    Run Keyword And Continue On Failure
-    ...    Wait Until Keyword Succeeds
-    ...    5 min
-    ...    10 sec
-    ...    Verify Openvino Deployment
-    ...    runtime_name=${runtime_pod_name}
-    ...    project_name=${PRJ_TITLE}
-    Run Keyword And Continue On Failure
-    ...    Wait Until Keyword Succeeds
-    ...    5 min
-    ...    10 sec
-    ...    Verify Serving Service
-    ...    project_name=${PRJ_TITLE}
-    Verify Model Status     ${MODEL_NAME}           success
-    Set Suite Variable      ${MODEL_CREATED}        ${TRUE}     # robocop: disable:replace-set-variable-with-var
-    Run Keyword And Continue On Failure
-    ...    Verify Model Inference
-    ...    ${MODEL_NAME}
-    ...    ${INFERENCE_INPUT_OPENVINO}
-    ...    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}
-    ...    token_auth=${FALSE}
-    Remove File     openshift_ca.crt
-    [Teardown]      Run Keywords        Dashboard Test Teardown
-    ...     AND
-    ...     Run Keyword If Test Failed      Get Events And Pod Logs     namespace=${PRJ_TITLE}
-    ...     label_selector=name=modelmesh-serving-${runtime_pod_name}
+    [Tags]                  Upgrade    ModelServing    ModelServer
+    Set Suite Variable    ${TEST_NS}    ovmsmodel-upgrade
+    Set Suite Variable    ${KSERVE_MODE}    RawDeployment    # RawDeployment   # Serverless
+    Set Suite Variable    ${MODELS_BUCKET}    ${S3.BUCKET_1}
path: ods_ci/tests/Tests/0200__rhoai_upgrade/0201__pre_upgrade.robot
user: login: github-advanced-security[bot]
body: ## {{ set_variable_keyword }} can be replaced with VAR

Set Suite Variable can be replaced with VAR

[Show more details](https://github.com/red-hat-data-services/ods-ci/security/code-scanning/28744)
created_at: 2025-04-01T08:46:49Z
author_association: NONE
diff_hunk: @@ -90,73 +90,46 @@
     [Tags]      Upgrade    ModelServing
     Disable Model Serving Runtime Using CLI     namespace=redhat-ods-applications
 
-Verify Model Can Be Deployed Via UI For Upgrade
+Verify Model Can Be Deployed For Upgrade
     # robocop: off=too-long-test-case
     # robocop: off=too-many-calls-in-test-case
     [Documentation]    Verify Model Can Be Deployed Via UI For Upgrade
-    [Tags]                  Upgrade    ModelServing
-    [Setup]                 Begin Web Test
-    ${PRJ_TITLE}=           Set Variable            model-serving-upgrade
-    ${PRJ_DESCRIPTION}=     Set Variable            project used for model serving tests
-    ${MODEL_NAME}=          Set Variable            test-model
-    ${MODEL_CREATED}=       Set Variable            ${FALSE}
-    ${RUNTIME_NAME}=        Set Variable            Model Serving Test
-    ${INFERENCE_INPUT_OPENVINO}=    Set Variable
-    ...    @tests/Resources/Files/openvino-example-input.json
-    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}=    Set Variable
-    ...    {"model_name":"test-model__isvc-8655dc7979","model_version":"1","outputs":[{"name":"Func/StatefulPartitionedCall/output/_13:0","datatype":"FP32","shape":[1,1],"data":[0.99999994]}]}   # robocop: disable:line-too-long
-    ${runtime_pod_name}=    Replace String Using Regexp
-    ...    string=${RUNTIME_NAME}
-    ...    pattern=\\s
-    ...    replace_with=-
-    ${runtime_pod_name}=    Convert To Lower Case                           ${runtime_pod_name}
-    Fetch CA Certificate If RHODS Is Self-Managed
-    Clean All Models Of Current User
-    Open Data Science Projects Home Page
-    Wait For RHODS Dashboard To Load
-    ...    wait_for_cards=${FALSE}
-    ...    expected_page=Data Science Projects
-    Create Data Science Project     title=${PRJ_TITLE}      description=${PRJ_DESCRIPTION}
-    Create S3 Data Connection
-    ...    project_title=${PRJ_TITLE}
-    ...    dc_name=model-serving-connection
-    ...    aws_access_key=${S3.AWS_ACCESS_KEY_ID}
-    ...    aws_secret_access=${S3.AWS_SECRET_ACCESS_KEY}
-    ...    aws_bucket_name=ods-ci-s3
-    Create Model Server     token=${FALSE}      server_name=${RUNTIME_NAME}
-    Serve Model
-    ...    project_name=${PRJ_TITLE}
-    ...    model_name=${MODEL_NAME}
-    ...    framework=openvino_ir
-    ...    existing_data_connection=${TRUE}
-    ...    data_connection_name=model-serving-connection
-    ...    model_path=openvino-example-model
-    Run Keyword And Continue On Failure
-    ...    Wait Until Keyword Succeeds
-    ...    5 min
-    ...    10 sec
-    ...    Verify Openvino Deployment
-    ...    runtime_name=${runtime_pod_name}
-    ...    project_name=${PRJ_TITLE}
-    Run Keyword And Continue On Failure
-    ...    Wait Until Keyword Succeeds
-    ...    5 min
-    ...    10 sec
-    ...    Verify Serving Service
-    ...    project_name=${PRJ_TITLE}
-    Verify Model Status     ${MODEL_NAME}           success
-    Set Suite Variable      ${MODEL_CREATED}        ${TRUE}     # robocop: disable:replace-set-variable-with-var
-    Run Keyword And Continue On Failure
-    ...    Verify Model Inference
-    ...    ${MODEL_NAME}
-    ...    ${INFERENCE_INPUT_OPENVINO}
-    ...    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}
-    ...    token_auth=${FALSE}
-    Remove File     openshift_ca.crt
-    [Teardown]      Run Keywords        Dashboard Test Teardown
-    ...     AND
-    ...     Run Keyword If Test Failed      Get Events And Pod Logs     namespace=${PRJ_TITLE}
-    ...     label_selector=name=modelmesh-serving-${runtime_pod_name}
+    [Tags]                  Upgrade    ModelServing    ModelServer
+    Set Suite Variable    ${TEST_NS}    ovmsmodel-upgrade
+    Set Suite Variable    ${KSERVE_MODE}    RawDeployment    # RawDeployment   # Serverless
+    Set Suite Variable    ${MODELS_BUCKET}    ${S3.BUCKET_1}
+    Set Suite Variable    ${INFERENCE_INPUT}    @tests/Resources/Files/modelmesh-mnist-input.json
path: ods_ci/tests/Tests/0200__rhoai_upgrade/0201__pre_upgrade.robot
user: login: github-advanced-security[bot]
body: ## {{ set_variable_keyword }} can be replaced with VAR

Set Suite Variable can be replaced with VAR

[Show more details](https://github.com/red-hat-data-services/ods-ci/security/code-scanning/28745)
created_at: 2025-04-01T08:46:49Z
author_association: NONE
diff_hunk: @@ -90,73 +90,46 @@
     [Tags]      Upgrade    ModelServing
     Disable Model Serving Runtime Using CLI     namespace=redhat-ods-applications
 
-Verify Model Can Be Deployed Via UI For Upgrade
+Verify Model Can Be Deployed For Upgrade
     # robocop: off=too-long-test-case
     # robocop: off=too-many-calls-in-test-case
     [Documentation]    Verify Model Can Be Deployed Via UI For Upgrade
-    [Tags]                  Upgrade    ModelServing
-    [Setup]                 Begin Web Test
-    ${PRJ_TITLE}=           Set Variable            model-serving-upgrade
-    ${PRJ_DESCRIPTION}=     Set Variable            project used for model serving tests
-    ${MODEL_NAME}=          Set Variable            test-model
-    ${MODEL_CREATED}=       Set Variable            ${FALSE}
-    ${RUNTIME_NAME}=        Set Variable            Model Serving Test
-    ${INFERENCE_INPUT_OPENVINO}=    Set Variable
-    ...    @tests/Resources/Files/openvino-example-input.json
-    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}=    Set Variable
-    ...    {"model_name":"test-model__isvc-8655dc7979","model_version":"1","outputs":[{"name":"Func/StatefulPartitionedCall/output/_13:0","datatype":"FP32","shape":[1,1],"data":[0.99999994]}]}   # robocop: disable:line-too-long
-    ${runtime_pod_name}=    Replace String Using Regexp
-    ...    string=${RUNTIME_NAME}
-    ...    pattern=\\s
-    ...    replace_with=-
-    ${runtime_pod_name}=    Convert To Lower Case                           ${runtime_pod_name}
-    Fetch CA Certificate If RHODS Is Self-Managed
-    Clean All Models Of Current User
-    Open Data Science Projects Home Page
-    Wait For RHODS Dashboard To Load
-    ...    wait_for_cards=${FALSE}
-    ...    expected_page=Data Science Projects
-    Create Data Science Project     title=${PRJ_TITLE}      description=${PRJ_DESCRIPTION}
-    Create S3 Data Connection
-    ...    project_title=${PRJ_TITLE}
-    ...    dc_name=model-serving-connection
-    ...    aws_access_key=${S3.AWS_ACCESS_KEY_ID}
-    ...    aws_secret_access=${S3.AWS_SECRET_ACCESS_KEY}
-    ...    aws_bucket_name=ods-ci-s3
-    Create Model Server     token=${FALSE}      server_name=${RUNTIME_NAME}
-    Serve Model
-    ...    project_name=${PRJ_TITLE}
-    ...    model_name=${MODEL_NAME}
-    ...    framework=openvino_ir
-    ...    existing_data_connection=${TRUE}
-    ...    data_connection_name=model-serving-connection
-    ...    model_path=openvino-example-model
-    Run Keyword And Continue On Failure
-    ...    Wait Until Keyword Succeeds
-    ...    5 min
-    ...    10 sec
-    ...    Verify Openvino Deployment
-    ...    runtime_name=${runtime_pod_name}
-    ...    project_name=${PRJ_TITLE}
-    Run Keyword And Continue On Failure
-    ...    Wait Until Keyword Succeeds
-    ...    5 min
-    ...    10 sec
-    ...    Verify Serving Service
-    ...    project_name=${PRJ_TITLE}
-    Verify Model Status     ${MODEL_NAME}           success
-    Set Suite Variable      ${MODEL_CREATED}        ${TRUE}     # robocop: disable:replace-set-variable-with-var
-    Run Keyword And Continue On Failure
-    ...    Verify Model Inference
-    ...    ${MODEL_NAME}
-    ...    ${INFERENCE_INPUT_OPENVINO}
-    ...    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}
-    ...    token_auth=${FALSE}
-    Remove File     openshift_ca.crt
-    [Teardown]      Run Keywords        Dashboard Test Teardown
-    ...     AND
-    ...     Run Keyword If Test Failed      Get Events And Pod Logs     namespace=${PRJ_TITLE}
-    ...     label_selector=name=modelmesh-serving-${runtime_pod_name}
+    [Tags]                  Upgrade    ModelServing    ModelServer
+    Set Suite Variable    ${TEST_NS}    ovmsmodel-upgrade
+    Set Suite Variable    ${KSERVE_MODE}    RawDeployment    # RawDeployment   # Serverless
+    Set Suite Variable    ${MODELS_BUCKET}    ${S3.BUCKET_1}
+    Set Suite Variable    ${INFERENCE_INPUT}    @tests/Resources/Files/modelmesh-mnist-input.json
+    Set Suite Variable    ${EXPECTED_INFERENCE_OUTPUT}    {"model_name": "test-dir","model_version": "1","outputs": [{"name": "Plus214_Output_0","shape": [1, 10],"datatype": "FP32","data": [-8.233053207397461, -7.749703407287598, -3.4236814975738527, 12.363029479980469, -12.079103469848633, 17.2665958404541, -10.570976257324219, 0.7130761742591858, 3.3217151165008547, 1.3621227741241456]}]}  #robocop: disable
+    Setup Test Variables    model_name=test-dir    use_pvc=${TRUE}    use_gpu=${FALSE}
+    ...    kserve_mode=${KSERVE_MODE}
+    Set Project And Runtime    runtime=ovms-runtime     protocol=http     namespace=${TEST_NS}
+    ...    download_in_pvc=${TRUE}    model_name=${model_name}
+    ...    storage_size=100Mi    memory_request=100Mi
+    ${requests}=    Create Dictionary    memory=1Gi
path: ods_ci/tests/Tests/0200__rhoai_upgrade/0201__pre_upgrade.robot
user: login: github-advanced-security[bot]
body: ## {{ create_keyword }} can be replaced with VAR

Create Dictionary can be replaced with VAR

[Show more details](https://github.com/red-hat-data-services/ods-ci/security/code-scanning/28746)
created_at: 2025-04-01T08:46:49Z
author_association: NONE
diff_hunk: @@ -156,30 +156,29 @@
 Test Inference Post RHODS Upgrade
     # robocop: off=too-many-calls-in-test-case
     # robocop: off=too-long-test-case
-    [Documentation]    Test the inference result after having deployed a model that requires Token Authentication
-    [Tags]                  Upgrade    ModelServing
-    [Setup]                 Begin Web Test
-    ${PRJ_TITLE}            Set Variable        model-serving-upgrade
-    ${PRJ_DESCRIPTION}      Set Variable        project used for model serving tests        # robocop: off=unused-variable      # robocop: disable:line-too-long
-    ${MODEL_NAME}           Set Variable        test-model
-    ${MODEL_CREATED}        Set Variable        ${FALSE}        # robocop: off=unused-variable
-    ${RUNTIME_NAME}         Set Variable        Model Serving Test      # robocop: off=unused-variable
-    ${INFERENCE_INPUT}      Set Variable        @tests/Resources/Files/modelmesh-mnist-input.json       # robocop: off=unused-variable      # robocop: disable:line-too-long
-    ${INFERENCE_INPUT_OPENVINO}    Set Variable
-    ...    @tests/Resources/Files/openvino-example-input.json
-    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}    Set Variable
-    ...    {"model_name":"test-model__isvc-8655dc7979","model_version":"1","outputs":[{"name":"Func/StatefulPartitionedCall/output/_13:0","datatype":"FP32","shape":[1,1],"data":[0.99999994]}]}   # robocop: disable:line-too-long
-    Fetch CA Certificate If RHODS Is Self-Managed
-    Open Model Serving Home Page
-    Verify Model Status     ${MODEL_NAME}           success
-    Run Keyword And Continue On Failure
-    ...    Verify Model Inference
-    ...    ${MODEL_NAME}
-    ...    ${INFERENCE_INPUT_OPENVINO}
-    ...    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}
-    ...    token_auth=${FALSE}
-    Remove File     openshift_ca.crt
-    [Teardown]      Run     oc delete project ${PRJ_TITLE}
+    [Documentation]    Test the inference result after having deployed a model
+    [Tags]                  Upgrade    ModelServing    ModelServer
+    Set Suite Variable    ${TEST_NS}    ovmsmodel-upgrade
path: ods_ci/tests/Tests/0200__rhoai_upgrade/0203__post_upgrade.robot
user: login: github-advanced-security[bot]
body: ## {{ set_variable_keyword }} can be replaced with VAR

Set Suite Variable can be replaced with VAR

[Show more details](https://github.com/red-hat-data-services/ods-ci/security/code-scanning/28737)
created_at: 2025-04-01T08:46:49Z
author_association: NONE
diff_hunk: @@ -156,30 +156,29 @@
 Test Inference Post RHODS Upgrade
     # robocop: off=too-many-calls-in-test-case
     # robocop: off=too-long-test-case
-    [Documentation]    Test the inference result after having deployed a model that requires Token Authentication
-    [Tags]                  Upgrade    ModelServing
-    [Setup]                 Begin Web Test
-    ${PRJ_TITLE}            Set Variable        model-serving-upgrade
-    ${PRJ_DESCRIPTION}      Set Variable        project used for model serving tests        # robocop: off=unused-variable      # robocop: disable:line-too-long
-    ${MODEL_NAME}           Set Variable        test-model
-    ${MODEL_CREATED}        Set Variable        ${FALSE}        # robocop: off=unused-variable
-    ${RUNTIME_NAME}         Set Variable        Model Serving Test      # robocop: off=unused-variable
-    ${INFERENCE_INPUT}      Set Variable        @tests/Resources/Files/modelmesh-mnist-input.json       # robocop: off=unused-variable      # robocop: disable:line-too-long
-    ${INFERENCE_INPUT_OPENVINO}    Set Variable
-    ...    @tests/Resources/Files/openvino-example-input.json
-    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}    Set Variable
-    ...    {"model_name":"test-model__isvc-8655dc7979","model_version":"1","outputs":[{"name":"Func/StatefulPartitionedCall/output/_13:0","datatype":"FP32","shape":[1,1],"data":[0.99999994]}]}   # robocop: disable:line-too-long
-    Fetch CA Certificate If RHODS Is Self-Managed
-    Open Model Serving Home Page
-    Verify Model Status     ${MODEL_NAME}           success
-    Run Keyword And Continue On Failure
-    ...    Verify Model Inference
-    ...    ${MODEL_NAME}
-    ...    ${INFERENCE_INPUT_OPENVINO}
-    ...    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}
-    ...    token_auth=${FALSE}
-    Remove File     openshift_ca.crt
-    [Teardown]      Run     oc delete project ${PRJ_TITLE}
+    [Documentation]    Test the inference result after having deployed a model
+    [Tags]                  Upgrade    ModelServing    ModelServer
+    Set Suite Variable    ${TEST_NS}    ovmsmodel-upgrade
+    Set Suite Variable    ${KSERVE_MODE}    RawDeployment    # RawDeployment   # Serverless
path: ods_ci/tests/Tests/0200__rhoai_upgrade/0203__post_upgrade.robot
user: login: github-advanced-security[bot]
body: ## {{ set_variable_keyword }} can be replaced with VAR

Set Suite Variable can be replaced with VAR

[Show more details](https://github.com/red-hat-data-services/ods-ci/security/code-scanning/28738)
created_at: 2025-04-01T08:46:49Z
author_association: NONE
diff_hunk: @@ -156,30 +156,29 @@
 Test Inference Post RHODS Upgrade
     # robocop: off=too-many-calls-in-test-case
     # robocop: off=too-long-test-case
-    [Documentation]    Test the inference result after having deployed a model that requires Token Authentication
-    [Tags]                  Upgrade    ModelServing
-    [Setup]                 Begin Web Test
-    ${PRJ_TITLE}            Set Variable        model-serving-upgrade
-    ${PRJ_DESCRIPTION}      Set Variable        project used for model serving tests        # robocop: off=unused-variable      # robocop: disable:line-too-long
-    ${MODEL_NAME}           Set Variable        test-model
-    ${MODEL_CREATED}        Set Variable        ${FALSE}        # robocop: off=unused-variable
-    ${RUNTIME_NAME}         Set Variable        Model Serving Test      # robocop: off=unused-variable
-    ${INFERENCE_INPUT}      Set Variable        @tests/Resources/Files/modelmesh-mnist-input.json       # robocop: off=unused-variable      # robocop: disable:line-too-long
-    ${INFERENCE_INPUT_OPENVINO}    Set Variable
-    ...    @tests/Resources/Files/openvino-example-input.json
-    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}    Set Variable
-    ...    {"model_name":"test-model__isvc-8655dc7979","model_version":"1","outputs":[{"name":"Func/StatefulPartitionedCall/output/_13:0","datatype":"FP32","shape":[1,1],"data":[0.99999994]}]}   # robocop: disable:line-too-long
-    Fetch CA Certificate If RHODS Is Self-Managed
-    Open Model Serving Home Page
-    Verify Model Status     ${MODEL_NAME}           success
-    Run Keyword And Continue On Failure
-    ...    Verify Model Inference
-    ...    ${MODEL_NAME}
-    ...    ${INFERENCE_INPUT_OPENVINO}
-    ...    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}
-    ...    token_auth=${FALSE}
-    Remove File     openshift_ca.crt
-    [Teardown]      Run     oc delete project ${PRJ_TITLE}
+    [Documentation]    Test the inference result after having deployed a model
+    [Tags]                  Upgrade    ModelServing    ModelServer
+    Set Suite Variable    ${TEST_NS}    ovmsmodel-upgrade
+    Set Suite Variable    ${KSERVE_MODE}    RawDeployment    # RawDeployment   # Serverless
+    Set Suite Variable    ${INFERENCE_INPUT}    @tests/Resources/Files/modelmesh-mnist-input.json
path: ods_ci/tests/Tests/0200__rhoai_upgrade/0203__post_upgrade.robot
user: login: github-advanced-security[bot]
body: ## {{ set_variable_keyword }} can be replaced with VAR

Set Suite Variable can be replaced with VAR

[Show more details](https://github.com/red-hat-data-services/ods-ci/security/code-scanning/28739)
created_at: 2025-04-01T08:46:49Z
author_association: NONE
diff_hunk: @@ -156,30 +156,29 @@
 Test Inference Post RHODS Upgrade
     # robocop: off=too-many-calls-in-test-case
     # robocop: off=too-long-test-case
-    [Documentation]    Test the inference result after having deployed a model that requires Token Authentication
-    [Tags]                  Upgrade    ModelServing
-    [Setup]                 Begin Web Test
-    ${PRJ_TITLE}            Set Variable        model-serving-upgrade
-    ${PRJ_DESCRIPTION}      Set Variable        project used for model serving tests        # robocop: off=unused-variable      # robocop: disable:line-too-long
-    ${MODEL_NAME}           Set Variable        test-model
-    ${MODEL_CREATED}        Set Variable        ${FALSE}        # robocop: off=unused-variable
-    ${RUNTIME_NAME}         Set Variable        Model Serving Test      # robocop: off=unused-variable
-    ${INFERENCE_INPUT}      Set Variable        @tests/Resources/Files/modelmesh-mnist-input.json       # robocop: off=unused-variable      # robocop: disable:line-too-long
-    ${INFERENCE_INPUT_OPENVINO}    Set Variable
-    ...    @tests/Resources/Files/openvino-example-input.json
-    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}    Set Variable
-    ...    {"model_name":"test-model__isvc-8655dc7979","model_version":"1","outputs":[{"name":"Func/StatefulPartitionedCall/output/_13:0","datatype":"FP32","shape":[1,1],"data":[0.99999994]}]}   # robocop: disable:line-too-long
-    Fetch CA Certificate If RHODS Is Self-Managed
-    Open Model Serving Home Page
-    Verify Model Status     ${MODEL_NAME}           success
-    Run Keyword And Continue On Failure
-    ...    Verify Model Inference
-    ...    ${MODEL_NAME}
-    ...    ${INFERENCE_INPUT_OPENVINO}
-    ...    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}
-    ...    token_auth=${FALSE}
-    Remove File     openshift_ca.crt
-    [Teardown]      Run     oc delete project ${PRJ_TITLE}
+    [Documentation]    Test the inference result after having deployed a model
+    [Tags]                  Upgrade    ModelServing    ModelServer
+    Set Suite Variable    ${TEST_NS}    ovmsmodel-upgrade
+    Set Suite Variable    ${KSERVE_MODE}    RawDeployment    # RawDeployment   # Serverless
+    Set Suite Variable    ${INFERENCE_INPUT}    @tests/Resources/Files/modelmesh-mnist-input.json
+    Set Suite Variable    ${EXPECTED_INFERENCE_OUTPUT}    {"model_name": "test-dir","model_version": "1","outputs": [{"name": "Plus214_Output_0","shape": [1, 10],"datatype": "FP32","data": [-8.233053207397461, -7.749703407287598, -3.4236814975738527, 12.363029479980469, -12.079103469848633, 17.2665958404541, -10.570976257324219, 0.7130761742591858, 3.3217151165008547, 1.3621227741241456]}]}  #robocop: disable
+    Setup Test Variables    model_name=test-dir    use_pvc=${TRUE}    use_gpu=${FALSE}
+    ...    kserve_mode=${KSERVE_MODE}
+    Wait For Pods To Be Ready    label_selector=serving.kserve.io/inferenceservice=${model_name}
+    ...    namespace=${TEST_NS}
+    ${pod_name}=  Get Pod Name    namespace=${TEST_NS}
path: ods_ci/tests/Tests/0200__rhoai_upgrade/0203__post_upgrade.robot
user: login: github-advanced-security[bot]
body: ## The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead

The assignment sign is not consistent within the file. Expected '' but got '=' instead

[Show more details](https://github.com/red-hat-data-services/ods-ci/security/code-scanning/28740)
created_at: 2025-04-01T08:46:49Z
author_association: NONE
diff_hunk: @@ -156,30 +156,29 @@
 Test Inference Post RHODS Upgrade
     # robocop: off=too-many-calls-in-test-case
     # robocop: off=too-long-test-case
-    [Documentation]    Test the inference result after having deployed a model that requires Token Authentication
-    [Tags]                  Upgrade    ModelServing
-    [Setup]                 Begin Web Test
-    ${PRJ_TITLE}            Set Variable        model-serving-upgrade
-    ${PRJ_DESCRIPTION}      Set Variable        project used for model serving tests        # robocop: off=unused-variable      # robocop: disable:line-too-long
-    ${MODEL_NAME}           Set Variable        test-model
-    ${MODEL_CREATED}        Set Variable        ${FALSE}        # robocop: off=unused-variable
-    ${RUNTIME_NAME}         Set Variable        Model Serving Test      # robocop: off=unused-variable
-    ${INFERENCE_INPUT}      Set Variable        @tests/Resources/Files/modelmesh-mnist-input.json       # robocop: off=unused-variable      # robocop: disable:line-too-long
-    ${INFERENCE_INPUT_OPENVINO}    Set Variable
-    ...    @tests/Resources/Files/openvino-example-input.json
-    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}    Set Variable
-    ...    {"model_name":"test-model__isvc-8655dc7979","model_version":"1","outputs":[{"name":"Func/StatefulPartitionedCall/output/_13:0","datatype":"FP32","shape":[1,1],"data":[0.99999994]}]}   # robocop: disable:line-too-long
-    Fetch CA Certificate If RHODS Is Self-Managed
-    Open Model Serving Home Page
-    Verify Model Status     ${MODEL_NAME}           success
-    Run Keyword And Continue On Failure
-    ...    Verify Model Inference
-    ...    ${MODEL_NAME}
-    ...    ${INFERENCE_INPUT_OPENVINO}
-    ...    ${EXPECTED_INFERENCE_OUTPUT_OPENVINO}
-    ...    token_auth=${FALSE}
-    Remove File     openshift_ca.crt
-    [Teardown]      Run     oc delete project ${PRJ_TITLE}
+    [Documentation]    Test the inference result after having deployed a model
+    [Tags]                  Upgrade    ModelServing    ModelServer
+    Set Suite Variable    ${TEST_NS}    ovmsmodel-upgrade
+    Set Suite Variable    ${KSERVE_MODE}    RawDeployment    # RawDeployment   # Serverless
+    Set Suite Variable    ${INFERENCE_INPUT}    @tests/Resources/Files/modelmesh-mnist-input.json
+    Set Suite Variable    ${EXPECTED_INFERENCE_OUTPUT}    {"model_name": "test-dir","model_version": "1","outputs": [{"name": "Plus214_Output_0","shape": [1, 10],"datatype": "FP32","data": [-8.233053207397461, -7.749703407287598, -3.4236814975738527, 12.363029479980469, -12.079103469848633, 17.2665958404541, -10.570976257324219, 0.7130761742591858, 3.3217151165008547, 1.3621227741241456]}]}  #robocop: disable
+    Setup Test Variables    model_name=test-dir    use_pvc=${TRUE}    use_gpu=${FALSE}
+    ...    kserve_mode=${KSERVE_MODE}
+    Wait For Pods To Be Ready    label_selector=serving.kserve.io/inferenceservice=${model_name}
+    ...    namespace=${TEST_NS}
+    ${pod_name}=  Get Pod Name    namespace=${TEST_NS}
+    ...    label_selector=serving.kserve.io/inferenceservice=${model_name}
+    ${service_port}=    Extract Service Port    service_name=${model_name}-predictor    protocol=TCP
path: ods_ci/tests/Tests/0200__rhoai_upgrade/0203__post_upgrade.robot
user: login: github-advanced-security[bot]
body: ## The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead

The assignment sign is not consistent within the file. Expected '' but got '=' instead

[Show more details](https://github.com/red-hat-data-services/ods-ci/security/code-scanning/28741)
created_at: 2025-04-01T08:46:49Z
author_association: NONE

PR comments:
user: login: rnetser
created_at: 2025-03-30T12:26:33Z
body: ```
14:29:49  Installing the current project: ods-ci (0.1.0)
14:29:53  ==============================================================================
14:29:53  Tests                                                                         
14:29:53  ==============================================================================
14:29:53  Tests.Rhoai Upgrade                                                           
14:29:53  ==============================================================================
14:29:57  2025-03-30 11:29:56,527 - RPA.core.certificates - INFO - Truststore not in use, HTTPS traffic validated against `certifi` package. (requires Python 3.10.12 and 'pip' 23.2.1 at minimum)
14:30:01  Tests.Rhoai Upgrade.Pre Upgrade :: Test Suite for Upgrade testing, to be ru...
14:30:01  ==============================================================================
14:30:02  [ WARN ] No Prometheus namespace found
14:30:12  Verify Model Can Be Deployed For Upgrade :: Verify Model Can Be De... status:PASS
14:30:17  status:PASS
14:30:27  Adding predictor model requests to tests/Resources/Files/llm/serving_runtimes/isvc/isvc_filled.yaml: {'memory': '1Gi'}
14:30:27  status:FAIL
14:30:29  status:PASS
14:30:51  | PASS |
14:30:51  ------------------------------------------------------------------------------
14:30:51  Tests.Rhoai Upgrade.Pre Upgrade :: Test Suite for Upgrade testing,... | PASS |
14:30:51  1 test, 1 passed, 0 failed
14:30:51  ==============================================================================
14:30:51  Tests.Rhoai Upgrade.Post Upgrade :: Test Suite for Upgrade testing,to be ru...
14:30:51  ==============================================================================
14:30:51  [ WARN ] No Prometheus namespace found
14:30:51  DataScienceCluster default-dsc in the namespace redhat-ods-operator attribute .status.release.name value is OpenShift AI Self-Managed
14:30:52  DSCInitialization default-dsci in the namespace redhat-ods-operator attribute .status.release.name value is OpenShift AI Self-Managed
14:30:53  DataScienceCluster default-dsc in the namespace redhat-ods-operator attribute .status.release.version value is 2.19.0
14:30:53  DSCInitialization default-dsci in the namespace redhat-ods-operator attribute .status.release.version value is 2.19.0
14:30:53  Test Inference Post RHODS Upgrade :: Test the inference result aft... status:PASS
14:31:20  | PASS |
14:31:20  ------------------------------------------------------------------------------
14:31:20  Tests.Rhoai Upgrade.Post Upgrade :: Test Suite for Upgrade testing... | PASS |
14:31:20  1 test, 1 passed, 0 failed
14:31:20  ==============================================================================
14:31:20  Tests.Rhoai Upgrade                                                   | PASS |
14:31:20  2 tests, 2 passed, 0 failed
14:31:20  ==============================================================================
14:31:20  Tests                                                                 | PASS |
14:31:20  2 tests, 2 passed, 0 failed
14:31:20  ==============================================================================
14:31:20  Output:  /home/jenkins/workspace/devops/rhoai-test-flow/ods-ci/ods_ci/test-output/ods-ci-2025-03-30-11-29-YmKNjJ8NSe/output.xml
14:31:20  XUnit:   /home/jenkins/workspace/devops/rhoai-test-flow/ods-ci/ods_ci/test-output/ods-ci-2025-03-30-11-29-YmKNjJ8NSe/xunit_test_result.xml
14:31:20  Log:     /home/jenkins/workspace/devops/rhoai-test-flow/ods-ci/ods_ci/test-output/ods-ci-2025-03-30-11-29-YmKNjJ8NSe/log.html
14:31:20  Report:  /home/jenkins/workspace/devops/rhoai-test-flow/ods-ci/ods_ci/test-output/ods-ci-2025-03-30-11-29-YmKNjJ8NSe/test_report.html
```
author_association: CONTRIBUTOR
user: login: rnetser
created_at: 2025-03-30T18:17:51Z
body: verified also as separate jobs
author_association: CONTRIBUTOR
user: login: sonarqubecloud[bot]
created_at: 2025-04-01T08:46:31Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=red-hat-data-services_ods-ci&pullRequest=2333) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=red-hat-data-services_ods-ci&pullRequest=2333&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=red-hat-data-services_ods-ci&pullRequest=2333&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=red-hat-data-services_ods-ci&pullRequest=2333&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=red-hat-data-services_ods-ci&pullRequest=2333&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=red-hat-data-services_ods-ci&pullRequest=2333&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=red-hat-data-services_ods-ci&pullRequest=2333)


author_association: NONE
user: login: openshift-ci[bot]
created_at: 2025-04-01T08:56:11Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

Approval requirements bypassed by manually added approval.

This pull-request has been approved by: *<a href="https://github.com/red-hat-data-services/ods-ci/pull/2333#pullrequestreview-2727891096" title="Approved">kobihk</a>*, *<a href="https://github.com/red-hat-data-services/ods-ci/pull/2333#pullrequestreview-2731955451" title="Approved">mwaykole</a>*, *<a href="https://github.com/red-hat-data-services/ods-ci/pull/2333#" title="Author self-approved">rnetser</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=red-hat-data-services%2Fods-ci).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/red-hat-data-services/ods-ci/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: NONE