Chapter 3. Serving models on the multi-model serving platform For deploying small and medium-sized models, OpenShift AI includes a multi-model serving platform that is based on the ModelMesh component. On the multi-model serving platform, multiple models can be deployed from the same model server and share the server resources. 3.1. Configuring model servers 3.1.1. Enabling the multi-model serving platform To use the multi-model serving platform, you must first enable the platform. Prerequisites You have logged in to OpenShift AI as a user with OpenShift AI administrator privileges. Your cluster administrator has not edited the OpenShift AI dashboard configuration to disable the ability to select the multi-model serving platform, which uses the ModelMesh component. For more information, see Dashboard configuration options . Procedure In the left menu of the OpenShift AI dashboard, click Settings Cluster settings . Locate the Model serving platforms section. Select the Multi-model serving platform checkbox. Click Save changes . 3.1.2. Adding a custom model-serving runtime for the multi-model serving platform A model-serving runtime adds support for a specified set of model frameworks and the model formats supported by those frameworks. By default, the multi-model serving platform includes the OpenVINO Model Server runtime. You can also add your own custom runtime if the default runtime does not meet your needs, such as supporting a specific model format. As an administrator, you can use the Red Hat OpenShift AI dashboard to add and enable a custom model-serving runtime. You can then choose the custom runtime when you create a new model server for the multi-model serving platform. Note Red Hat does not provide support for custom runtimes. You are responsible for ensuring that you are licensed to use any custom runtimes that you add, and for correctly configuring and maintaining them. Prerequisites You have logged in to OpenShift AI as a user with OpenShift AI administrator privileges. You are familiar with how to add a model server to your project . When you have added a custom model-serving runtime, you must configure a new model server to use the runtime. You have reviewed the example runtimes in the kserve/modelmesh-serving repository. You can use these examples as starting points. However, each runtime requires some further modification before you can deploy it in OpenShift AI. The required modifications are described in the following procedure. Note OpenShift AI includes the OpenVINO Model Server runtime by default. You do not need to add this runtime to OpenShift AI. Procedure From the OpenShift AI dashboard, click Settings > Serving runtimes . The Serving runtimes page opens and shows the model-serving runtimes that are already installed and enabled. To add a custom runtime, choose one of the following options: To start with an existing runtime (for example the OpenVINO Model Server runtime), click the action menu (...) to the existing runtime and then click Duplicate . To add a new custom runtime, click Add serving runtime . In the Select the model serving platforms this runtime supports list, select Multi-model serving platform . Note The multi-model serving platform supports only the REST protocol. Therefore, you cannot change the default value in the Select the API protocol this runtime supports list. Optional: If you started a new runtime (rather than duplicating an existing one), add your code by choosing one of the following options: Upload a YAML file Click Upload files . In the file browser, select a YAML file on your computer. This file might be the one of the example runtimes that you downloaded from the kserve/modelmesh-serving repository. The embedded YAML editor opens and shows the contents of the file that you uploaded. Enter YAML code directly in the editor Click Start from scratch . Enter or paste YAML code directly in the embedded editor. The YAML that you paste might be copied from one of the example runtimes in the kserve/modelmesh-serving repository. Optional: If you are adding one of the example runtimes in the kserve/modelmesh-serving repository, perform the following modifications: In the YAML editor, locate the kind field for your runtime. Update the value of this field to ServingRuntime . In the kustomization.yaml file in the kserve/modelmesh-serving repository, take note of the newName and newTag values for the runtime that you want to add. You will specify these values in a later step. In the YAML editor for your custom runtime, locate the containers.image field. Update the value of the containers.image field in the format newName:newTag , based on the values that you previously noted in the kustomization.yaml file. Some examples are shown. Nvidia Triton Inference Server image: nvcr.io/nvidia/tritonserver:23.04-py3 Seldon Python MLServer image: seldonio/mlserver:1.3.2 TorchServe image: pytorch/torchserve:0.7.1-cpu In the metadata.name field, ensure that the value of the runtime you are adding is unique (that is, the value doesn't match a runtime that you have already added). Optional: To configure a custom display name for the runtime that you are adding, add a metadata.annotations.openshift.io/display-name field and specify a value, as shown in the following example: Note If you do not configure a custom display name for your runtime, OpenShift AI shows the value of the metadata.name field. Click Add . The Serving runtimes page opens and shows the updated list of runtimes that are installed. Observe that the runtime you added is automatically enabled. Optional: To edit your custom runtime, click the action menu (...) and select Edit . Verification The custom model-serving runtime that you added is shown in an enabled state on the Serving runtimes page. Additional resources To learn how to configure a model server that uses a custom model-serving runtime that you have added, see Adding a model server to your data science project . 3.1.3. Adding a tested and verified model-serving runtime for the multi-model serving platform In addition to preinstalled and custom model-serving runtimes, you can also use Red Hat tested and verified model-serving runtimes such as the NVIDIA Triton Inference Server to support your needs. For more information about Red Hat tested and verified runtimes, see Tested and verified runtimes for Red Hat OpenShift AI . You can use the Red Hat OpenShift AI dashboard to add and enable the NVIDIA Triton Inference Server runtime and then choose the runtime when you create a new model server for the multi-model serving platform. Prerequisites You have logged in to OpenShift AI as a user with OpenShift AI administrator privileges. You are familiar with how to add a model server to your project . After you have added a tested and verified model-serving runtime, you must configure a new model server to use the runtime. Procedure From the OpenShift AI dashboard, click Settings > Serving runtimes . The Serving runtimes page opens and shows the model-serving runtimes that are already installed and enabled. To add a tested and verified runtime, click Add serving runtime . In the Select the model serving platforms this runtime supports list, select Multi-model serving platform . Note The multi-model serving platform supports only the REST protocol. Therefore, you cannot change the default value in the Select the API protocol this runtime supports list. Click Start from scratch . Enter or paste the following YAML code directly in the embedded editor. In the metadata.name field, make sure that the value of the runtime you are adding does not match a runtime that you have already added). Optional: To use a custom display name for the runtime that you are adding, add a metadata.annotations.openshift.io/display-name field and specify a value, as shown in the following example: Note If you do not configure a custom display name for your runtime, OpenShift AI shows the value of the metadata.name field. Click Create . The Serving runtimes page opens and shows the updated list of runtimes that are installed. Observe that the runtime you added is automatically enabled. Optional: To edit the runtime, click the action menu (...) and select Edit . Verification The model-serving runtime that you added is shown in an enabled state on the Serving runtimes page. Additional resources To learn how to configure a model server that uses a model-serving runtime that you have added, see Adding a model server to your data science project . 3.1.4. Adding a model server for the multi-model serving platform When you have enabled the multi-model serving platform, you must configure a model server to deploy models. If you require extra computing power for use with large datasets, you can assign accelerators to your model server. Note In OpenShift AI, Red Hat supports only NVIDIA and AMD GPU accelerators for model serving. Prerequisites You have logged in to Red Hat OpenShift AI. If you use OpenShift AI groups, you are part of the user group or admin group (for example, rhoai-users or rhoai-admins ) in OpenShift. You have created a data science project that you can add a model server to. You have enabled the multi-model serving platform. If you want to use a custom model-serving runtime for your model server, you have added and enabled the runtime. See Adding a custom model-serving runtime . If you want to use graphics processing units (GPUs) with your model server, you have enabled GPU support in OpenShift AI. If you use NVIDIA GPUs, see Enabling NVIDIA GPUs . If you use AMD GPUs, see AMD GPU integration . Procedure In the left menu of the OpenShift AI dashboard, click Data Science Projects . The Data Science Projects page opens. Click the name of the project that you want to configure a model server for. A project details page opens. Click the Models tab. Perform one of the following actions: If you see a Multi-model serving platform tile, click Add model server on the tile. If you do not see any tiles, click the Add model server button. The Add model server dialog opens. In the Model server name field, enter a unique name for the model server. From the Serving runtime list, select a model-serving runtime that is installed and enabled in your OpenShift AI deployment. Note If you are using a custom model-serving runtime with your model server and want to use GPUs, you must ensure that your custom runtime supports GPUs and is appropriately configured to use them. In the Number of model replicas to deploy field, specify a value. From the Model server size list, select a value. Optional: If you selected Custom in the preceding step, configure the following settings in the Model server size section to customize your model server: In the CPUs requested field, specify the number of CPUs to use with your model server. Use the list beside this field to specify the value in cores or millicores. In the CPU limit field, specify the maximum number of CPUs to use with your model server. Use the list beside this field to specify the value in cores or millicores. In the Memory requested field, specify the requested memory for the model server in gibibytes (Gi). In the Memory limit field, specify the maximum memory limit for the model server in gibibytes (Gi). Optional: From the Accelerator list, select an accelerator. If you selected an accelerator in the preceding step, specify the number of accelerators to use. Optional: In the Model route section, select the Make deployed models available through an external route checkbox to make your deployed models available to external clients. Optional: In the Token authentication section, select the Require token authentication checkbox to require token authentication for your model server. To finish configuring token authentication, perform the following actions: In the Service account name field, enter a service account name for which the token will be generated. The generated token is created and displayed in the Token secret field when the model server is configured. To add an additional service account, click Add a service account and enter another service account name. Click Add . The model server that you configured appears on the Models tab for the project, in the Models and model servers list. Optional: To update the model server, click the action menu ( ... ) beside the model server and select Edit model server . 3.1.5. Deleting a model server When you no longer need a model server to host models, you can remove it from your data science project. Note When you remove a model server, you also remove the models that are hosted on that model server. As a result, the models are no longer available to applications. Prerequisites You have created a data science project and an associated model server. You have notified the users of the applications that access the models that the models will no longer be available. If you are using OpenShift AI groups, you are part of the user group or admin group (for example, rhoai-users or rhoai-admins ) in OpenShift. Procedure From the OpenShift AI dashboard, click Data Science Projects . The Data Science Projects page opens. Click the name of the project from which you want to delete the model server. A project details page opens. Click the Models tab. Click the action menu ( ... ) beside the project whose model server you want to delete and then click Delete model server . The Delete model server dialog opens. Enter the name of the model server in the text field to confirm that you intend to delete it. Click Delete model server . Verification The model server that you deleted is no longer displayed on the Models tab for the project. 3.2. Working with deployed models 3.2.1. Deploying a model by using the multi-model serving platform You can deploy trained models on OpenShift AI to enable you to test and implement them into intelligent applications. Deploying a model makes it available as a service that you can access by using an API. This enables you to return predictions based on data inputs. When you have enabled the multi-model serving platform, you can deploy models on the platform. Prerequisites You have logged in to Red Hat OpenShift AI. If you are using OpenShift AI groups, you are part of the user group or admin group (for example, rhoai-users ) in OpenShift. You have enabled the multi-model serving platform. You have created a data science project and added a model server. You have access to S3-compatible object storage. For the model that you want to deploy, you know the associated folder path in your S3-compatible object storage bucket. Procedure In the left menu of the OpenShift AI dashboard, click Data Science Projects . The Data Science Projects page opens. Click the name of the project that you want to deploy a model in. A project details page opens. Click the Models tab. Click Deploy model . Configure properties for deploying your model as follows: In the Model name field, enter a unique name for the model that you are deploying. From the Model framework list, select a framework for your model. Note The Model framework list shows only the frameworks that are supported by the model-serving runtime that you specified when you configured your model server. To specify the location of the model you want to deploy from S3-compatible object storage, perform one of the following sets of actions: To use an existing connection Select Existing connection . From the Name list, select a connection that you previously defined. In the Path field, enter the folder path that contains the model in your specified data source. To use a new connection To define a new connection that your model can access, select New connection . In the Add connection modal, select a Connection type . The S3 compatible object storage and URI options are pre-installed connection types. Additional options might be available if your OpenShift AI administrator added them. The Add connection form opens with fields specific to the connection type that you selected. Enter the connection detail fields. (Optional) Customize the runtime parameters in the Configuration parameters section: Modify the values in Additional serving runtime arguments to define how the deployed model behaves. Modify the values in Additional environment variables to define variables in the model's environment. Click Deploy . Verification Confirm that the deployed model is shown on the Models tab for the project, and on the Model Serving page of the dashboard with a checkmark in the Status column. 3.2.2. Viewing a deployed model To analyze the results of your work, you can view a list of deployed models on Red Hat OpenShift AI. You can also view the current statuses of deployed models and their endpoints. Prerequisites You have logged in to Red Hat OpenShift AI. If you are using OpenShift AI groups, you are part of the user group or admin group (for example, rhoai-users or rhoai-admins ) in OpenShift. Procedure From the OpenShift AI dashboard, click Model Serving . The Deployed models page opens. For each model, the page shows details such as the model name, the project in which the model is deployed, the model-serving runtime that the model uses, and the deployment status. Optional: For a given model, click the link in the Inference endpoint column to see the inference endpoints for the deployed model. Verification A list of previously deployed data science models is displayed on the Deployed models page. 3.2.3. Updating the deployment properties of a deployed model You can update the deployment properties of a model that has been deployed previously. For example, you can change the model's connection and name. Prerequisites You have logged in to Red Hat OpenShift AI. If you are using OpenShift AI groups, you are part of the user group or admin group (for example, rhoai-users or rhoai-admins ) in OpenShift. You have deployed a model on OpenShift AI. Procedure From the OpenShift AI dashboard, click Model Serving . The Deployed models page opens. Click the action menu ( ... ) beside the model whose deployment properties you want to update and click Edit . The Edit model dialog opens. Update the deployment properties of the model as follows: In the Model name field, enter a new, unique name for your model. From the Model servers list, select a model server for your model. From the Model framework list, select a framework for your model. Note The Model framework list shows only the frameworks that are supported by the model-serving runtime that you specified when you configured your model server. Optionally, update the connection by specifying an existing connection or by creating a new connection. Click Redeploy . Verification The model whose deployment properties you updated is displayed on the Model Serving page of the dashboard. 3.2.4. Deleting a deployed model You can delete models you have previously deployed. This enables you to remove deployed models that are no longer required. Prerequisites You have logged in to Red Hat OpenShift AI. If you are using OpenShift AI groups, you are part of the user group or admin group (for example, rhoai-users or rhoai-admins ) in OpenShift. You have deployed a model. Procedure From the OpenShift AI dashboard, click Model serving . The Deployed models page opens. Click the action menu ( ... ) beside the deployed model that you want to delete and click Delete . The Delete deployed model dialog opens. Enter the name of the deployed model in the text field to confirm that you intend to delete it. Click Delete deployed model . Verification The model that you deleted is no longer displayed on the Deployed models page. 3.3. Viewing model-serving runtime metrics for the multi-model serving platform After a cluster administrator has configured monitoring for the multi-model serving platform, non-admin users can use the OpenShift web console to view model-serving runtime metrics for the ModelMesh component. Prerequisites You have access to the OpenShift cluster as a developer or as a user with view permissions for the project that you are viewing metrics for. You are familiar with querying metrics in user-defined projects. See Monitoring project and application metrics using the Developer perspective in Red Hat OpenShift Dedicated or Monitoring project and application metrics using the Developer perspective in Red Hat OpenShift Service on AWS . Procedure Log in to the OpenShift web console. Switch to the Developer perspective. In the left menu, click Observe . As described in Monitoring your project metrics in Red Hat OpenShift Dedicated or Monitoring your project metrics in Red Hat OpenShift Service on AWS , use the web console to run queries for modelmesh_* metrics. 3.4. Monitoring model performance In the multi-model serving platform, you can view performance metrics for all models deployed on a model server and for a specific model that is deployed on the model server. 3.4.1. Viewing performance metrics for all models on a model server You can monitor the following metrics for all the models that are deployed on a model server: HTTP requests per 5 minutes - The number of HTTP requests that have failed or succeeded for all models on the server. Average response time (ms) - For all models on the server, the average time it takes the model server to respond to requests. CPU utilization (%) - The percentage of the CPU's capacity that is currently being used by all models on the server. Memory utilization (%) - The percentage of the system's memory that is currently being used by all models on the server. You can specify a time range and a refresh interval for these metrics to help you determine, for example, when the peak usage hours are and how the models are performing at a specified time. Prerequisites You have installed Red Hat OpenShift AI. On the OpenShift cluster where OpenShift AI is installed, user workload monitoring is enabled. You have logged in to Red Hat OpenShift AI. If you are using OpenShift AI groups, you are part of the user group or admin group (for example, rhoai-users or rhoai-admins ) in OpenShift. You have deployed models on the multi-model serving platform. Procedure From the OpenShift AI dashboard navigation menu, click Data Science Projects . The Data Science Projects page opens. Click the name of the project that contains the data science models that you want to monitor. In the project details page, click the Models tab. In the row for the model server that you are interested in, click the action menu (...) and then select View model server metrics . Optional: On the metrics page for the model server, set the following options: Time range - Specifies how long to track the metrics. You can select one of these values: 1 hour, 24 hours, 7 days, and 30 days. Refresh interval - Specifies how frequently the graphs on the metrics page are refreshed (to show the latest data). You can select one of these values: 15 seconds, 30 seconds, 1 minute, 5 minutes, 15 minutes, 30 minutes, 1 hour, 2 hours, and 1 day. Scroll down to view data graphs for HTTP requests per 5 minutes, average response time, CPU utilization, and memory utilization. Verification On the metrics page for the model server, the graphs provide data on performance metrics. 3.4.2. Viewing HTTP request metrics for a deployed model You can view a graph that illustrates the HTTP requests that have failed or succeeded for a specific model that is deployed on the multi-model serving platform. Prerequisites You have installed Red Hat OpenShift AI. On the OpenShift cluster where OpenShift AI is installed, user workload monitoring is enabled. The following dashboard configuration options are set to the default values as shown: For more information, see Dashboard configuration options . You have logged in to Red Hat OpenShift AI. If you are using OpenShift AI groups, you are part of the user group or admin group (for example, rhoai-users or rhoai-admins ) in OpenShift. You have deployed models on the multi-model serving platform. Procedure From the OpenShift AI dashboard navigation menu, select Model Serving . On the Deployed models page, select the model that you are interested in. Optional: On the Endpoint performance tab, set the following options: Time range - Specifies how long to track the metrics. You can select one of these values: 1 hour, 24 hours, 7 days, and 30 days. Refresh interval - Specifies how frequently the graphs on the metrics page are refreshed (to show the latest data). You can select one of these values: 15 seconds, 30 seconds, 1 minute, 5 minutes, 15 minutes, 30 minutes, 1 hour, 2 hours, and 1 day. Verification The Endpoint performance tab shows a graph of the HTTP metrics for the model.

Commands:
apiVersion: serving.kserve.io/v1alpha1 kind: ServingRuntime metadata: name: mlserver-0.x annotations: openshift.io/display-name: MLServer
apiVersion: serving.kserve.io/v1alpha1 kind: ServingRuntime metadata: annotations: enable-route: "true" name: modelmesh-triton labels: opendatahub.io/dashboard: "true" spec: annotations: opendatahub.io/modelServingSupport: '["multi"x`x`]' prometheus.kserve.io/path: /metrics prometheus.kserve.io/port: "8002" builtInAdapter: env: - name: CONTAINER_MEM_REQ_BYTES value: "268435456" - name: USE_EMBEDDED_PULLER value: "true" memBufferBytes: 134217728 modelLoadingTimeoutMillis: 90000 runtimeManagementPort: 8001 serverType: triton containers: - args: - -c - 'mkdir -p /models/_triton_models; chmod 777 /models/_triton_models; exec tritonserver "--model-repository=/models/_triton_models" "--model-control-mode=explicit" "--strict-model-config=false" "--strict-readiness=false" "--allow-http=true" "--allow-grpc=true" ' command: - /bin/sh image: nvcr.io/nvidia/tritonserver@sha256:xxxxx name: triton resources: limits: cpu: "1" memory: 2Gi requests: cpu: "1" memory: 2Gi grpcDataEndpoint: port:8001 grpcEndpoint: port:8085 multiModel: true protocolVersions: - grpc-v2 - v2 supportedModelFormats: - autoSelect: true name: onnx version: "1" - autoSelect: true name: pytorch version: "1" - autoSelect: true name: tensorflow version: "1" - autoSelect: true name: tensorflow version: "2" - autoSelect: true name: tensorrt version: "7" - autoSelect: false name: xgboost version: "1" - autoSelect: true name: python version: "1"
apiVersion: serving.kserve.io/v1alpha1 kind: ServingRuntime metadata: name: modelmesh-triton annotations: openshift.io/display-name: Triton ServingRuntime
disablePerformanceMetrics:false disableKServeMetrics:false