Chapter 2. Serving models on the single-model serving platform For deploying large models such as large language models (LLMs), Red Hat OpenShift AI includes a single model serving platform that is based on the KServe component. Because each model is deployed from its own model server, the single model serving platform helps you to deploy, monitor, scale, and maintain large models that require increased resources. 2.1. About the single-model serving platform For deploying large models such as large language models (LLMs), OpenShift AI includes a single-model serving platform that is based on the KServe component. Because each model is deployed on its own model server, the single-model serving platform helps you to deploy, monitor, scale, and maintain large models that require increased resources. 2.2. Components KServe : A Kubernetes custom resource definition (CRD) that orchestrates model serving for all types of models. KServe includes model-serving runtimes that implement the loading of given types of model servers. KServe also handles the lifecycle of the deployment object, storage access, and networking setup. Red Hat OpenShift Serverless : A cloud-native development model that allows for serverless deployments of models. OpenShift Serverless is based on the open source Knative project. Red Hat OpenShift Service Mesh : A service mesh networking layer that manages traffic flows and enforces access policies. OpenShift Service Mesh is based on the open source Istio project. 2.3. Installation options To install the single-model serving platform, you have the following options: Automated installation If you have not already created a ServiceMeshControlPlane or KNativeServing resource on your OpenShift cluster, you can configure the Red Hat OpenShift AI Operator to install KServe and configure its dependencies. For more information about automated installation, see Configuring automated installation of KServe . Manual installation If you have already created a ServiceMeshControlPlane or KNativeServing resource on your OpenShift cluster, you cannot configure the Red Hat OpenShift AI Operator to install KServe and configure its dependencies. In this situation, you must install KServe manually. For more information about manual installation, see Manually installing KServe . 2.4. Authorization You can add Authorino as an authorization provider for the single-model serving platform. Adding an authorization provider allows you to enable token authentication for models that you deploy on the platform, which ensures that only authorized parties can make inference requests to the models. To add Authorino as an authorization provider on the single-model serving platform, you have the following options: If automated installation of the single-model serving platform is possible on your cluster, you can include Authorino as part of the automated installation process. If you need to manually install the single-model serving platform, you must also manually configure Authorino. For guidance on choosing an installation option for the single-model serving platform, see Installation options . 2.5. Monitoring You can configure monitoring for the single-model serving platform and use Prometheus to scrape metrics for each of the pre-installed model-serving runtimes. 2.6. Model-serving runtimes You can serve models on the single-model serving platform by using model-serving runtimes. The configuration of a model-serving runtime is defined by the ServingRuntime and InferenceService custom resource definitions (CRDs). 2.6.1. ServingRuntime The ServingRuntime CRD creates a serving runtime, an environment for deploying and managing a model. It creates the templates for pods that dynamically load and unload models of various formats and also exposes a service endpoint for inferencing requests. The following YAML configuration is an example of the vLLM ServingRuntime for KServe model-serving runtime. The configuration includes various flags, environment variables and command-line arguments. 1 The recommended accelerator to use with the runtime. 2 The name with which the serving runtime is displayed. 3 The endpoint used by Prometheus to scrape metrics for monitoring. 4 The port used by Prometheus to scrape metrics for monitoring. 5 The path to where the model files are stored in the runtime container. 6 Passes the model name that is specified by the {{.Name}} template variable inside the runtime container specification to the runtime environment. The {{.Name}} variable maps to the spec.predictor.name field in the InferenceService metadata object. 7 The entrypoint command that starts the runtime container. 8 The runtime container image used by the serving runtime. This image differs depending on the type of accelerator used. 9 Specifies that the runtime is used for single-model serving. 10 Specifies the model formats supported by the runtime. 2.6.2. InferenceService The InferenceService CRD creates a server or inference service that processes inference queries, passes it to the model, and then returns the inference output. The inference service also performs the following actions: Specifies the location and format of the model. Specifies the serving runtime used to serve the model. Enables the passthrough route for gRPC or REST inference. Defines HTTP or gRPC endpoints for the deployed model. The following example shows the InferenceService YAML configuration file that is generated when deploying a granite model with the vLLM runtime: Additional resources Serving Runtimes 2.7. Supported model-serving runtimes OpenShift AI includes several preinstalled model-serving runtimes. You can use preinstalled model-serving runtimes to start serving models without modifying or defining the runtime yourself. You can also add a custom runtime to support a model. For help adding a custom runtime, see Adding a custom model-serving runtime for the single-model serving platform . Table 2.1. Model-serving runtimes Name Description Exported model format Caikit Text Generation Inference Server (Caikit-TGIS) ServingRuntime for KServe (1) A composite runtime for serving models in the Caikit format Caikit Text Generation Caikit Standalone ServingRuntime for KServe (2) A runtime for serving models in the Caikit embeddings format for embeddings tasks Caikit Embeddings OpenVINO Model Server A scalable, high-performance runtime for serving models that are optimized for Intel architectures PyTorch, TensorFlow, OpenVINO IR, PaddlePaddle, MXNet, Caffe, Kaldi Text Generation Inference Server (TGIS) Standalone ServingRuntime for KServe (3) A runtime for serving TGI-enabled models PyTorch Model Formats vLLM ServingRuntime for KServe A high-throughput and memory-efficient inference and serving runtime for large language models Supported models vLLM ServingRuntime with Gaudi accelerators support for KServe A high-throughput and memory-efficient inference and serving runtime that supports Intel Gaudi accelerators Supported models vLLM ROCm ServingRuntime for KServe A high-throughput and memory-efficient inference and serving runtime that supports AMD GPU accelerators Supported models The composite Caikit-TGIS runtime is based on Caikit and Text Generation Inference Server (TGIS) . To use this runtime, you must convert your models to Caikit format. For an example, see Converting Hugging Face Hub models to Caikit format in the caikit-tgis-serving repository. The Caikit Standalone runtime is based on Caikit NLP . To use this runtime, you must convert your models to the Caikit embeddings format. For an example, see Tests for text embedding module . Text Generation Inference Server (TGIS) is based on an early fork of Hugging Face TGI . Red Hat will continue to develop the standalone TGIS runtime to support TGI models. If a model is incompatible in the current version of OpenShift AI, support might be added in a future version. In the meantime, you can also add your own custom runtime to support a TGI model. For more information, see Adding a custom model-serving runtime for the single-model serving platform . Table 2.2. Deployment requirements Name Default protocol Additonal protocol Model mesh support Single node OpenShift support Deployment mode Caikit Text Generation Inference Server (Caikit-TGIS) ServingRuntime for KServe REST gRPC No Yes Raw and serverless Caikit Standalone ServingRuntime for KServe REST gRPC No Yes Raw and serverless OpenVINO Model Server REST None Yes Yes Raw and serverless Text Generation Inference Server (TGIS) Standalone ServingRuntime for KServe gRPC None No Yes Raw and serverless vLLM ServingRuntime for KServe REST None No Yes Raw and serverless vLLM ServingRuntime with Gaudi accelerators support for KServe REST None No Yes Raw and serverless vLLM ROCm ServingRuntime for KServe REST None No Yes Raw and serverless Additional resources Inference endpoints 2.8. Tested and verified model-serving runtimes Tested and verified runtimes are community versions of model-serving runtimes that have been tested and verified against specific versions of OpenShift AI. Red Hat tests the current version of a tested and verified runtime each time there is a new version of OpenShift AI. If a new version of a tested and verified runtime is released in the middle of an OpenShift AI release cycle, it will be tested and verified in an upcoming release. A list of the tested and verified runtimes and compatible versions is available in the OpenShift AI release notes . Note Tested and verified runtimes are not directly supported by Red Hat. You are responsible for ensuring that you are licensed to use any tested and verified runtimes that you add, and for correctly configuring and maintaining them. For more information, see Tested and verified runtimes in OpenShift AI . Table 2.3. Model-serving runtimes Name Description Exported model format NVIDIA Triton Inference Server An open-source inference-serving software for fast and scalable AI in applications. TensorRT, TensorFlow, PyTorch, ONNX, OpenVINO, Python, RAPIDS FIL, and more Table 2.4. Deployment requirements Name Default protocol Additonal protocol Model mesh support Single node OpenShift support Deployment mode NVIDIA Triton Inference Server gRPC REST Yes Yes Raw and serverless Additional resources Inference endpoints 2.9. Inference endpoints These examples show how to use inference endpoints to query the model. Note If you enabled token authentication when deploying the model, add the Authorization header and specify a token value. 2.9.1. Caikit TGIS ServingRuntime for KServe :443/api/v1/task/text-generation :443/api/v1/task/server-streaming-text-generation Example command 2.9.2. Caikit Standalone ServingRuntime for KServe If you are serving multiple models, you can query /info/models or :443 caikit.runtime.info.InfoService/GetModelsInfo to view a list of served models. REST endpoints /api/v1/task/embedding /api/v1/task/embedding-tasks /api/v1/task/sentence-similarity /api/v1/task/sentence-similarity-tasks /api/v1/task/rerank /api/v1/task/rerank-tasks /info/models /info/version /info/runtime gRPC endpoints :443 caikit.runtime.Nlp.NlpService/EmbeddingTaskPredict :443 caikit.runtime.Nlp.NlpService/EmbeddingTasksPredict :443 caikit.runtime.Nlp.NlpService/SentenceSimilarityTaskPredict :443 caikit.runtime.Nlp.NlpService/SentenceSimilarityTasksPredict :443 caikit.runtime.Nlp.NlpService/RerankTaskPredict :443 caikit.runtime.Nlp.NlpService/RerankTasksPredict :443 caikit.runtime.info.InfoService/GetModelsInfo :443 caikit.runtime.info.InfoService/GetRuntimeInfo Note By default, the Caikit Standalone Runtime exposes REST endpoints. To use gRPC protocol, manually deploy a custom Caikit Standalone ServingRuntime. For more information, see Adding a custom model-serving runtime for the single-model serving platform . An example manifest is available in the caikit-tgis-serving GitHub repository . REST gRPC 2.9.3. TGIS Standalone ServingRuntime for KServe :443 fmaas.GenerationService/Generate :443 fmaas.GenerationService/GenerateStream Note To query the endpoint for the TGIS standalone runtime, you must also download the files in the proto directory of the OpenShift AI text-generation-inference repository. Example command 2.9.4. OpenVINO Model Server /v2/models/<model-name>/infer Example command 2.9.5. vLLM ServingRuntime for KServe :443/version :443/docs :443/v1/models :443/v1/chat/completions :443/v1/completions :443/v1/embeddings :443/tokenize :443/detokenize Note The vLLM runtime is compatible with the OpenAI REST API. For a list of models that the vLLM runtime supports, see Supported models . To use the embeddings inference endpoint in vLLM, you must use an embeddings model that the vLLM supports. You cannot use the embeddings endpoint with generative models. For more information, see Supported embeddings models in vLLM . As of vLLM v0.5.5, you must provide a chat template while querying a model using the /v1/chat/completions endpoint. If your model does not include a predefined chat template, you can use the chat-template command-line parameter to specify a chat template in your custom vLLM runtime, as shown in the example. Replace <CHAT_TEMPLATE> with the path to your template. You can use the chat templates that are available as .jinja files here or with the vLLM image under /apps/data/template . For more information, see Chat templates . As indicated by the paths shown, the single-model serving platform uses the HTTPS port of your OpenShift router (usually port 443) to serve external API requests. Example command 2.9.6. vLLM ServingRuntime with Gaudi accelerators support for KServe See vLLM ServingRuntime for KServe . 2.9.7. vLLM ROCm ServingRuntime for KServe See vLLM ServingRuntime for KServe . 2.9.8. NVIDIA Triton Inference Server REST endpoints v2/models/[/versions/<model_version>]/infer v2/models/<model_name>[/versions/<model_version>] v2/health/ready v2/health/live v2/models/<model_name>[/versions/]/ready v2 Note ModelMesh does not support the following REST endpoints: v2/health/live v2/health/ready v2/models/<model_name>[/versions/]/ready Example command gRPC endpoints :443 inference.GRPCInferenceService/ModelInfer :443 inference.GRPCInferenceService/ModelReady :443 inference.GRPCInferenceService/ModelMetadata :443 inference.GRPCInferenceService/ServerReady :443 inference.GRPCInferenceService/ServerLive :443 inference.GRPCInferenceService/ServerMetadata Example command 2.9.9. Additional resources Text Generation Inference Server (TGIS) Caikit API documentation Caikit NLP GitHub project OpenVINO KServe-compatible REST API documentation OpenAI API documentation Open Inference Protocol Supported model-serving runtimes . 2.10. About KServe deployment modes By default, you can deploy models on the single-model serving platform with KServe by using Red Hat OpenShift Serverless , which is a cloud-native development model that allows for serverless deployments of models. OpenShift Serverless is based on the open source Knative project. In addition, serverless mode is dependent on the Red Hat OpenShift Serverless Operator. Alternatively, you can use raw deployment mode, which is not dependent on the Red Hat OpenShift Serverless Operator. With raw deployment mode, you can deploy models with Kubernetes resources, such as Deployment , Service , Ingress , and Horizontal Pod Autoscaler . Important Deploying a machine learning model using KServe raw deployment mode is a Limited Availability feature. Limited Availability means that you can install and receive support for the feature only with specific approval from the Red Hat AI Business Unit. Without such approval, the feature is unsupported. In addition, this feature is only supported on Self-Managed deployments of single node OpenShift. There are both advantages and disadvantages to using each of these deployment modes: 2.10.1. Serverless mode Advantages: Enables autoscaling based on request volume: Resources scale up automatically when receiving incoming requests. Optimizes resource usage and maintains performance during peak times. Supports scale down to and from zero using Knative: Allows resources to scale down completely when there are no incoming requests. Saves costs by not running idle resources. Disadvantages: Has customization limitations: Serverless is limited to Knative, such as when mounting multiple volumes. Dependency on Knative for scaling: Introduces additional complexity in setup and management compared to traditional scaling methods. 2.10.2. Raw deployment mode Advantages: Enables deployment with Kubernetes resources, such as Deployment , Service , Ingress , and Horizontal Pod Autoscaler : Provides full control over Kubernetes resources, allowing for detailed customization and configuration of deployment settings. Unlocks Knative limitations, such as being unable to mount multiple volumes: Beneficial for applications requiring complex configurations or multiple storage mounts. Disadvantages: Does not support automatic scaling: Does not support automatic scaling down to zero resources when idle. Might result in higher costs during periods of low traffic. Requires manual management of scaling. 2.11. Deploying models by using the single-model serving platform On the single-model serving platform, each model is deployed on its own model server. This helps you to deploy, monitor, scale, and maintain large models that require increased resources. Important If you want to use the single-model serving platform to deploy a model from S3-compatible storage that uses a self-signed SSL certificate, you must install a certificate authority (CA) bundle on your OpenShift cluster. For more information, see Working with certificates . 2.11.1. Enabling the single-model serving platform When you have installed KServe, you can use the Red Hat OpenShift AI dashboard to enable the single-model serving platform. You can also use the dashboard to enable model-serving runtimes for the platform. Prerequisites You have logged in to OpenShift AI as a user with OpenShift AI administrator privileges. You have installed KServe. Your cluster administrator has not edited the OpenShift AI dashboard configuration to disable the ability to select the single-model serving platform, which uses the KServe component. For more information, see Dashboard configuration options . Procedure Enable the single-model serving platform as follows: In the left menu, click Settings Cluster settings . Locate the Model serving platforms section. To enable the single-model serving platform for projects, select the Single-model serving platform checkbox. Click Save changes . Enable preinstalled runtimes for the single-model serving platform as follows: In the left menu of the OpenShift AI dashboard, click Settings Serving runtimes . The Serving runtimes page shows preinstalled runtimes and any custom runtimes that you have added. For more information about preinstalled runtimes, see Supported runtimes . Set the runtime that you want to use to Enabled . The single-model serving platform is now available for model deployments. 2.11.2. Adding a custom model-serving runtime for the single-model serving platform A model-serving runtime adds support for a specified set of model frameworks and the model formats supported by those frameworks. You can use the pre-installed runtimes that are included with OpenShift AI. You can also add your own custom runtimes if the default runtimes do not meet your needs. For example, if the TGIS runtime does not support a model format that is supported by Hugging Face Text Generation Inference (TGI) , you can create a custom runtime to add support for the model. As an administrator, you can use the OpenShift AI interface to add and enable a custom model-serving runtime. You can then choose the custom runtime when you deploy a model on the single-model serving platform. Note Red Hat does not provide support for custom runtimes. You are responsible for ensuring that you are licensed to use any custom runtimes that you add, and for correctly configuring and maintaining them. Prerequisites You have logged in to OpenShift AI as a user with OpenShift AI administrator privileges. You have built your custom runtime and added the image to a container image repository such as Quay . Procedure From the OpenShift AI dashboard, click Settings > Serving runtimes . The Serving runtimes page opens and shows the model-serving runtimes that are already installed and enabled. To add a custom runtime, choose one of the following options: To start with an existing runtime (for example, TGIS Standalone ServingRuntime for KServe ), click the action menu (...) to the existing runtime and then click Duplicate . To add a new custom runtime, click Add serving runtime . In the Select the model serving platforms this runtime supports list, select Single-model serving platform . In the Select the API protocol this runtime supports list, select REST or gRPC . Optional: If you started a new runtime (rather than duplicating an existing one), add your code by choosing one of the following options: Upload a YAML file Click Upload files . In the file browser, select a YAML file on your computer. The embedded YAML editor opens and shows the contents of the file that you uploaded. Enter YAML code directly in the editor Click Start from scratch . Enter or paste YAML code directly in the embedded editor. Note In many cases, creating a custom runtime will require adding new or custom parameters to the env section of the ServingRuntime specification. Click Add . The Serving runtimes page opens and shows the updated list of runtimes that are installed. Observe that the custom runtime that you added is automatically enabled. The API protocol that you specified when creating the runtime is shown. Optional: To edit your custom runtime, click the action menu (...) and select Edit . Verification The custom model-serving runtime that you added is shown in an enabled state on the Serving runtimes page. 2.11.3. Adding a tested and verified model-serving runtime for the single-model serving platform In addition to preinstalled and custom model-serving runtimes, you can also use Red Hat tested and verified model-serving runtimes such as the NVIDIA Triton Inference Server to support your needs. For more information about Red Hat tested and verified runtimes, see Tested and verified runtimes for Red Hat OpenShift AI . You can use the Red Hat OpenShift AI dashboard to add and enable the NVIDIA Triton Inference Server runtime for the single-model serving platform. You can then choose the runtime when you deploy a model on the single-model serving platform. Prerequisites You have logged in to OpenShift AI as a user with OpenShift AI administrator privileges. Procedure From the OpenShift AI dashboard, click Settings > Serving runtimes . The Serving runtimes page opens and shows the model-serving runtimes that are already installed and enabled. Click Add serving runtime . In the Select the model serving platforms this runtime supports list, select Single-model serving platform . In the Select the API protocol this runtime supports list, select REST or gRPC . Click Start from scratch . If you selected the REST API protocol, enter or paste the following YAML code directly in the embedded editor. If you selected the gRPC API protocol, enter or paste the following YAML code directly in the embedded editor. In the metadata.name field, make sure that the value of the runtime you are adding does not match a runtime that you have already added). Optional: To use a custom display name for the runtime that you are adding, add a metadata.annotations.openshift.io/display-name field and specify a value, as shown in the following example: Note If you do not configure a custom display name for your runtime, OpenShift AI shows the value of the metadata.name field. Click Create . The Serving runtimes page opens and shows the updated list of runtimes that are installed. Observe that the runtime that you added is automatically enabled. The API protocol that you specified when creating the runtime is shown. Optional: To edit the runtime, click the action menu (...) and select Edit . Verification The model-serving runtime that you added is shown in an enabled state on the Serving runtimes page. Additional resources Tested and verified model-serving runtimes 2.11.4. Deploying models on the single-model serving platform When you have enabled the single-model serving platform, you can enable a preinstalled or custom model-serving runtime and start to deploy models on the platform. Note Text Generation Inference Server (TGIS) is based on an early fork of Hugging Face TGI . Red Hat will continue to develop the standalone TGIS runtime to support TGI models. If a model does not work in the current version of OpenShift AI, support might be added in a future version. In the meantime, you can also add your own, custom runtime to support a TGI model. For more information, see Adding a custom model-serving runtime for the single-model serving platform . Prerequisites You have logged in to Red Hat OpenShift AI. If you are using OpenShift AI groups, you are part of the user group or admin group (for example, rhoai-users or rhoai-admins ) in OpenShift. You have installed KServe. You have enabled the single-model serving platform. To enable token authentication and external model routes for deployed models, you have added Authorino as an authorization provider. For more information, see Adding an authorization provider for the single-model serving platform . You have created a data science project. You have access to S3-compatible object storage. For the model that you want to deploy, you know the associated folder path in your S3-compatible object storage bucket. To use the Caikit-TGIS runtime, you have converted your model to Caikit format. For an example, see Converting Hugging Face Hub models to Caikit format in the caikit-tgis-serving repository. If you want to use graphics processing units (GPUs) with your model server, you have enabled GPU support in OpenShift AI. If you use NVIDIA GPUs, see Enabling NVIDIA GPUs . If you use AMD GPUs, see AMD GPU integration . To use the vLLM runtime, you have enabled GPU support in OpenShift AI and have installed and configured the Node Feature Discovery operator on your cluster. For more information, see Installing the Node Feature Discovery operator and Enabling NVIDIA GPUs To use the vLLM ServingRuntime with Gaudi accelerators support for KServe runtime, you have enabled support for hybrid processing units (HPUs) in OpenShift AI. This includes installing the Intel Gaudi AI accelerator operator and configuring an accelerator profile. For more information, see Setting up Gaudi for OpenShift and Working with accelerators . To use the vLLM ROCm ServingRuntime for KServe runtime, you have enabled support for AMD graphic processing units (GPUs) in OpenShift AI. This includes installing the AMD GPU operator and configuring an accelerator profile. For more information, see Deploying the AMD GPU operator on OpenShift and Working with accelerators . Note In OpenShift AI, Red Hat supports NVIDIA GPU, Intel Gaudi, and AMD GPU accelerators for model serving. To deploy RHEL AI models: You have enabled the vLLM ServingRuntime for KServe runtime. You have downloaded the model from the Red Hat container registry and uploaded it to S3-compatible object storage. Procedure In the left menu, click Data Science Projects . The Data Science Projects page opens. Click the name of the project that you want to deploy a model in. A project details page opens. Click the Models tab. Perform one of the following actions: If you see a Single-model serving platform tile, click Deploy model on the tile. If you do not see any tiles, click the Deploy model button. The Deploy model dialog opens. In the Model deployment name field, enter a unique name for the model that you are deploying. In the Serving runtime field, select an enabled runtime. From the Model framework (name - version) list, select a value. In the Number of model server replicas to deploy field, specify a value. From the Model server size list, select a value. The following options are only available if you have enabled accelerator support on your cluster and created an accelerator profile: From the Accelerator list, select an accelerator. If you selected an accelerator in the preceding step, specify the number of accelerators to use in the Number of accelerators field. Optional: In the Model route section, select the Make deployed models available through an external route checkbox to make your deployed models available to external clients. To require token authentication for inference requests to the deployed model, perform the following actions: Select Require token authentication . In the Service account name field, enter the service account name that the token will be generated for. To add an additional service account, click Add a service account and enter another service account name. To specify the location of your model, perform one of the following sets of actions: To use an existing connection Select Existing connection . From the Name list, select a connection that you previously defined. In the Path field, enter the folder path that contains the model in your specified data source. Important The OpenVINO Model Server runtime has specific requirements for how you specify the model path. For more information, see known issue RHOAIENG-3025 in the OpenShift AI release notes. To use a new connection To define a new connection that your model can access, select New connection . In the Add connection modal, select a Connection type . The S3 compatible object storage and URI options are pre-installed connection types. Additional options might be available if your OpenShift AI administrator added them. The Add connection form opens with fields specific to the connection type that you selected. Fill in the connection detail fields. Important If your connection type is an S3-compatible object storage, you must provide the folder path that contains your data file. The OpenVINO Model Server runtime has specific requirements for how you specify the model path. For more information, see known issue RHOAIENG-3025 in the OpenShift AI release notes. (Optional) Customize the runtime parameters in the Configuration parameters section: Modify the values in Additional serving runtime arguments to define how the deployed model behaves. Modify the values in Additional environment variables to define variables in the model's environment. The Configuration parameters section shows predefined serving runtime parameters, if any are available. Note Do not modify the port or model serving runtime arguments, because they require specific values to be set. Overwriting these parameters can cause the deployment to fail. Click Deploy . Verification Confirm that the deployed model is shown on the Models tab for the project, and on the Model Serving page of the dashboard with a checkmark in the Status column. 2.11.5. Setting a timeout for KServe When deploying large models or using node autoscaling with KServe, the operation may time out before a model is deployed because the default progress-deadline that KNative Serving sets is 10 minutes. If a pod using KNative Serving takes longer than 10 minutes to deploy, the pod might be automatically marked as failed. This can happen if you are deploying large models that take longer than 10 minutes to pull from S3-compatible object storage or if you are using node autoscaling to reduce the consumption of GPU nodes. To resolve this issue, you can set a custom progress-deadline in the KServe InferenceService for your application. Prerequisites You have namespace edit access for your OpenShift cluster. Procedure Log in to the OpenShift console as a cluster administrator. Select the project where you have deployed the model. In the Administrator perspective, click Home Search . From the Resources dropdown menu, search for InferenceService . Under spec.predictor.annotations , modify the serving.knative.dev/progress-deadline with the new timeout: Note Ensure that you set the progress-deadline on the spec.predictor.annotations level, so that the KServe InferenceService can copy the progress-deadline back to the KNative Service object. 2.11.6. Customizing the parameters of a deployed model-serving runtime You might need additional parameters beyond the default ones to deploy specific models or to enhance an existing model deployment. In such cases, you can modify the parameters of an existing runtime to suit your deployment needs. Note Customizing the parameters of a runtime only affects the selected model deployment. Prerequisites You have logged in to OpenShift AI as a user with OpenShift AI administrator privileges. You have deployed a model on the single-model serving platform. Procedure From the OpenShift AI dashboard, click Model Serving in the left menu. The Deployed models page opens. Click the action menu (...) to the name of the model you want to customize and select Edit . The Configuration parameters section shows predefined serving runtime parameters, if any are available. Customize the runtime parameters in the Configuration parameters section: Modify the values in Additional serving runtime arguments to define how the deployed model behaves. Modify the values in Additional environment variables to define variables in the model's environment. Note Do not modify the port or model serving runtime arguments, because they require specific values to be set. Overwriting these parameters can cause the deployment to fail. After you are done customizing the runtime parameters, click Redeploy to save and deploy the model with your changes. Verification Confirm that the deployed model is shown on the Models tab for the project, and on the Model Serving page of the dashboard with a checkmark in the Status column. Confirm that the arguments and variables that you set appear in spec.predictor.model.args and spec.predictor.model.env by one of the following methods: Checking the InferenceService YAML from the OpenShift Console. Using the following command in the OpenShift CLI: 2.11.7. Customizable model serving runtime parameters You can modify the parameters of an existing model serving runtime to suit your deployment needs. For more information about parameters for each of the supported serving runtimes, see the following table: Serving runtime Resource NVIDIA Triton Inference Server NVIDIA Triton Inference Server: Model Parameters Caikit Text Generation Inference Server (Caikit-TGIS) ServingRuntime for KServe Caikit NLP: Configuration TGIS: Model configuration Caikit Standalone ServingRuntime for KServe Caikit NLP: Configuration OpenVINO Model Server OpenVINO Model Server Features: Dynamic Input Parameters Text Generation Inference Server (TGIS) Standalone ServingRuntime for KServe TGIS: Model configuration vLLM ServingRuntime for KServe vLLM: Engine Arguments OpenAI Compatible Server Additional resources Customizing the parameters of a deployed model serving runtime 2.11.8. Using OCI containers for model storage As an alternative to storing a model in an S3 bucket or URI, you can upload models to Open Container Initiative (OCI) containers. Using OCI containers for model storage can help you: Reduce startup times by avoiding downloading the same model multiple times. Reduce disk space usage by reducing the number of models downloaded locally. Improve model performance by allowing pre-fetched images. Using OCI containers for model storage involves the following tasks: Storing a model in an OCI image Deploying a model from an OCI image Important Using OCI containers for model storage is currently available in Red Hat OpenShift AI as a Technology Preview feature. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . 2.11.8.1. Storing a model in an OCI image You can store a model in an OCI image. The following procedure uses the example of storing a MobileNet v2-7 model in ONNX format. Prerequisites You have a model in the ONNX format. The example in this procedure uses the MobileNet v2-7 model in ONNX format. You have installed the Podman tool. Procedure In a terminal window on your local machine, create a temporary directory for storing both the model and the support files that you need to create the OCI image: Create a models folder inside the temporary directory: Note This example command specifies the subdirectory 1 because OpenVINO requires numbered subdirectories for model versioning. If you are not using OpenVINO, you do not need to create the 1 subdirectory to use OCI container images. Download the model and support files: Use the tree command to confirm that the model files are located in the directory structure as expected: The tree command should return a directory structure similar to the following example: Create a Docker file named Containerfile : Note Specify a base image that provides a shell. In the following example, ubi9-micro is the base container image. You cannot specify an empty image that does not provide a shell, such as scratch , because KServe uses the shell to ensure the model files are accessible to the model server. Change the ownership of the copied model files and grant read permissions to the root group to ensure that the model server can access the files. OpenShift runs containers with a random user ID and the root group ID. Use podman build commands to create the OCI container image and upload it to a registry. The following commands use Quay as the registry. Note If your repository is private, ensure that you are authenticated to the registry before uploading your container image. 2.11.8.2. Deploying a model stored in an OCI image You can deploy a model that is stored in an OCI image. The following procedure uses the example of deploying a MobileNet v2-7 model in ONNX format, stored in an OCI image on an OpenVINO model server. Note By default in KServe, models are exposed outside the cluster and not protected with authentication. Prerequisites You have stored a model in an OCI image as described in Storing a model in an OCI image . If you want to deploy a model that is stored in a private OCI repository, you must configure an image pull secret. For more information about creating an image pull secret, see Using image pull secrets . You are logged in to your OpenShift cluster. Procedure Create a project to deploy the model: Use the OpenShift AI Applications project kserve-ovms template to create a ServingRuntime resource and configure the OpenVINO model server in the new project: Verify that the ServingRuntime named kserve-ovms is created: The command should return output similar to the following: Create an InferenceService YAML resource, depending on whether the model is stored from a private or a public OCI repository: For a model stored in a public OCI repository, create an InferenceService YAML file with the following values, replacing <user_name> , <repository_name> , and <tag_name> with values specific to your environment: For a model stored in a private OCI repository, create an InferenceService YAML file that specifies your pull secret in the spec.predictor.imagePullSecrets field, as shown in the following example: After you create the InferenceService resource, KServe deploys the model stored in the OCI image referred to by the storageUri field. Verification Check the status of the deployment: The command should return output that includes information, such as the URL of the deployed model and its readiness state. 2.11.9. Using accelerators with vLLM OpenShift AI includes support for NVIDIA, AMD and Intel Gaudi accelerators. OpenShift AI also includes preinstalled model-serving runtimes that provide accelerator support. 2.11.9.1. NVIDIA GPUs You can serve models with NVIDIA graphics processing units (GPUs) by using the vLLM ServingRuntime for KServe runtime. To use the runtime, you must enable GPU support in OpenShift AI. This includes installing and configuring the Node Feature Discovery operator on your cluster. For more information, see Installing the Node Feature Discovery operator and Enabling NVIDIA GPUs . 2.11.9.2. Intel Gaudi accelerators You can serve models with Intel Gaudi accelerators by using the vLLM ServingRuntime with Gaudi accelerators support for KServe runtime. To use the runtime, you must enable hybrid processing support (HPU) support in OpenShift AI. This includes installing the Intel Gaudi AI accelerator operator and configuring an accelerator profile. For more information, see Setting up Gaudi for OpenShift and Working with accelerator profiles . For information about recommended vLLM parameters, environment variables, supported configurations and more, see vLLM with Intel(R) Gaudi(R) AI Accelerators . Note Warm-up is a model initialization and performance optimization step that is useful for reducing cold-start delays and first-inference latency. Depending on the model size, warm-up can lead to longer model loading times. While highly recommended in production environments to avoid performance limitations, you can choose to skip warm-up for non-production environments to reduce model loading times and accelerate model development and testing cycles. To skip warm-up, follow the steps described in Customizing the parameters of a deployed model-serving runtime to add the following environment variable in the Configuration parameters section of your model deployment: 2.11.9.3. AMD GPUs You can serve models with AMD GPUs by using the vLLM ROCm ServingRuntime for KServe runtime. To use the runtime, you must enable support for AMD graphic processing units (GPUs) in OpenShift AI. This includes installing the AMD GPU operator and configuring an accelerator profile. For more information, see Deploying the AMD GPU operator on OpenShift and Working with accelerator profiles . Additional resources Supported model-serving runtimes 2.11.10. Customizing the vLLM model-serving runtime In certain cases, you may need to add additional flags or environment variables to the vLLM ServingRuntime for KServe runtime to deploy a family of LLMs. The following procedure describes customizing the vLLM model-serving runtime to deploy a Llama, Granite or Mistral model. Prerequisites You have logged in to OpenShift AI as a user with OpenShift AI administrator privileges. For Llama model deployment, you have downloaded a meta-llama-3 model to your object storage. For Granite model deployment, you have downloaded a granite-7b-instruct or granite-20B-code-instruct model to your object storage. For Mistral model deployment, you have downloaded a mistral-7B-Instruct-v0.3 model to your object storage. You have enabled the vLLM ServingRuntime for KServe runtime. You have enabled GPU support in OpenShift AI and have installed and configured the Node Feature Discovery operator on your cluster. For more information, see Installing the Node Feature Discovery operator and Enabling NVIDIA GPUs Procedure Follow the steps to deploy a model as described in Deploying models on the single-model serving platform . In the Serving runtime field, select vLLM ServingRuntime for KServe . If you are deploying a meta-llama-3 model, add the following arguments under Additional serving runtime arguments in the Configuration parameters section: 1 Sets the backend to multiprocessing for distributed model workers 2 Sets the maximum context length of the model to 6144 tokens If you are deploying a granite-7B-instruct model, add the following arguments under Additional serving runtime arguments in the Configuration parameters section: 1 Sets the backend to multiprocessing for distributed model workers If you are deploying a granite-20B-code-instruct model, add the following arguments under Additional serving runtime arguments in the Configuration parameters section: 1 Sets the backend to multiprocessing for distributed model workers 2 Distributes inference across 4 GPUs in a single node 3 Sets the maximum context length of the model to 6448 tokens If you are deploying a mistral-7B-Instruct-v0.3 model, add the following arguments under Additional serving runtime arguments in the Configuration parameters section: 1 Sets the backend to multiprocessing for distributed model workers 2 Sets the maximum context length of the model to 15344 tokens Click Deploy . Verification Confirm that the deployed model is shown on the Models tab for the project, and on the Model Serving page of the dashboard with a checkmark in the Status column. For granite models, use the following example command to verify API requests to your deployed model: Additional resources vLLM: Engine Arguments 2.12. Deploying models by using multiple GPU nodes Deploy models across multiple GPU nodes to handle large models, such as large language models (LLMs). This procedure shows you how to serve models on Red Hat OpenShift AI across multiple GPU nodes using the vLLM serving framework. Multi-node inferencing uses the vllm-multinode-runtime custom runtime. The vllm-multinode-runtime runtime uses the same image as the VLLM ServingRuntime for KServe runtime and also includes information necessary for multi-GPU inferencing. Important Deploying models by using multiple GPU nodes is currently available in Red Hat OpenShift AI as a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope Prerequisites You have cluster administrator privileges for your OpenShift cluster. You have downloaded and installed the OpenShift command-line interface (CLI). For more information, see Installing the OpenShift CLI (Red Hat OpenShift Dedicated) or Installing the OpenShift CLI (Red Hat OpenShift Service on AWS). You have enabled the operators for your GPU type, such as Node Feature Discovery Operator, NVIDIA GPU Operator. For more information about enabling accelerators, see Enabling accelerators . You are using an NVIDIA GPU ( nvidia.com/gpu ). You have specified the GPU type through either the ServingRuntime or InferenceService . If the GPU type specified in the ServingRuntime differs from what is set in the InferenceService , both GPU types are assigned to the resource and can cause errors. You have enabled KServe on your cluster. You have only one head pod in your setup. Do not adjust the replica count using the min_replicas or max_replicas settings in the InferenceService . Creating additional head pods can cause them to be excluded from the Ray cluster. You have a persistent volume claim (PVC) set up and configured for ReadWriteMany (RWX) access mode. Procedure In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI as shown in the following example: Select or create a namespace for deploying the model. For example, you can create the kserve-demo namespace by running the following command: From the namespace where you would like to deploy the model, create a PVC for model storage. Create a storage class using Filesystem volumeMode . Use this storage class for your PVC. The storage size must be larger than the size of the model files on disk. For example: Note If you have already configured a PVC, you can skip this step. To download the model to the PVC, modify the sample YAML provided: 1 The chmod operation is permitted only if your pod is running as root. Remove`chmod -R 777` from the arguments if you are not running the pod as root. 2 7 Specify the path to the model. 3 The value for containers.image , located in your InferenceService . To access this value, run the following command: oc get configmap inferenceservice-config -n redhat-ods-operator -oyaml | grep kserve-storage-initializer: 4 The access key ID to your S3 bucket. 5 The secret access key to your S3 bucket. 6 The name of your S3 bucket. 8 The endpoint to your S3 bucket. 9 The region for your S3 bucket if using an AWS S3 bucket. If using other S3-compatible storage, such as ODF or Minio, you can remove the AWS_DEFAULT_REGION environment variable. 10 If you encounter SSL errors, change S3_VERIFY_SSL to false . Create the vllm-multinode-runtime custom runtime: Deploy the model using the following InferenceService configuration: The following configuration can be added to the InferenceService : workerSpec.tensorParallelSize : Determines how many GPUs are used per node. The GPU type count in both the head and worker node deployment resources is updated automatically. Ensure that the value of workerSpec.tensorParallelSize is at least 1 . workerSpec.pipelineParallelSize : Determines how many nodes are used to balance the model in deployment. This variable represents the total number of nodes, including both the head and worker nodes. Ensure that the value of workerSpec.pipelineParallelSize is at least 2 . Do not modify this value in production environments. Note You may need to specify additional arguments, depending on your environment and model size. Verification To confirm that you have set up your environment to deploy models on multiple GPU nodes, check the GPU resource status, the InferenceService status, the ray cluster status, and send a request to the model. Check the GPU resource status: Retrieve the pod names for the head and worker nodes: Sample response Confirm that the model loaded properly by checking the values of <1> and <2>. If the model did not load, the value of these fields is 0MiB . Verify the status of your InferenceService using the following command: NOTE: In the Technology Preview, you can only use port forwarding for inferencing. Sample response Send a request to the model to confirm that the model is available for inference: 2.13. Making inference requests to models deployed on the single-model serving platform When you deploy a model by using the single-model serving platform, the model is available as a service that you can access using API requests. This enables you to return predictions based on data inputs. To use API requests to interact with your deployed model, you must know the inference endpoint for the model. In addition, if you secured your inference endpoint by enabling token authentication, you must know how to access your authentication token so that you can specify this in your inference requests. 2.13.1. Accessing the authentication token for a deployed model If you secured your model inference endpoint by enabling token authentication, you must know how to access your authentication token so that you can specify it in your inference requests. Prerequisites You have logged in to Red Hat OpenShift AI. If you are using OpenShift AI groups, you are part of the user group or admin group (for example, rhoai-users or rhoai-admins ) in OpenShift. You have deployed a model by using the single-model serving platform. Procedure From the OpenShift AI dashboard, click Data Science Projects . The Data Science Projects page opens. Click the name of the project that contains your deployed model. A project details page opens. Click the Models tab. In the Models and model servers list, expand the section for your model. Your authentication token is shown in the Token authentication section, in the Token secret field. Optional: To copy the authentication token for use in an inference request, click the Copy button ( ) to the token value. 2.13.2. Accessing the inference endpoint for a deployed model To make inference requests to your deployed model, you must know how to access the inference endpoint that is available. For a list of paths to use with the supported runtimes and example commands, see Inference endpoints . Prerequisites You have logged in to Red Hat OpenShift AI. If you are using OpenShift AI groups, you are part of the user group or admin group (for example, rhoai-users or rhoai-admins ) in OpenShift. You have deployed a model by using the single-model serving platform. If you enabled token authentication for your deployed model, you have the associated token value. Procedure From the OpenShift AI dashboard, click Model Serving . The inference endpoint for the model is shown in the Inference endpoint field. Depending on what action you want to perform with the model (and if the model supports that action), copy the inference endpoint and then add a path to the end of the URL. Use the endpoint to make API requests to your deployed model. Additional resources Text Generation Inference Server (TGIS) Caikit API documentation Caikit NLP GitHub project OpenVINO KServe-compatible REST API documentation OpenAI API documentation 2.14. Viewing model-serving runtime metrics for the single-model serving platform When a cluster administrator has configured monitoring for the single-model serving platform, non-admin users can use the OpenShift web console to view model-serving runtime metrics for the KServe component. Prerequisites You have access to the OpenShift cluster as a developer or as a user with view permissions for the project that you are viewing metrics for. You are familiar with querying metrics in user-defined projects. See Monitoring project and application metrics using the Developer perspective in Red Hat OpenShift Dedicated or Monitoring project and application metrics using the Developer perspective in Red Hat OpenShift Service on AWS . Procedure Log in to the OpenShift web console. Switch to the Developer perspective. In the left menu, click Observe . As described in Monitoring your project metrics in Red Hat OpenShift Dedicated or Monitoring your project metrics in Red Hat OpenShift Service on AWS , use the web console to run queries for caikit_* , tgi_* , ovms_* and vllm:* model-serving runtime metrics. You can also run queries for istio_* metrics that are related to OpenShift Service Mesh. Some examples are shown. The following query displays the number of successful inference requests over a period of time for a model deployed with the vLLM runtime: The following query displays the number of successful inference requests over a period of time for a model deployed with the standalone TGIS runtime: The following query displays the number of successful inference requests over a period of time for a model deployed with the Caikit Standalone runtime: The following query displays the number of successful inference requests over a period of time for a model deployed with the OpenVINO Model Server runtime: Additional resources OVMS metrics TGIS metrics vLLM metrics 2.15. Monitoring model performance In the single-model serving platform, you can view performance metrics for a specific model that is deployed on the platform. 2.15.1. Viewing performance metrics for a deployed model You can monitor the following metrics for a specific model that is deployed on the single-model serving platform: Number of requests - The number of requests that have failed or succeeded for a specific model. Average response time (ms) - The average time it takes a specific model to respond to requests. CPU utilization (%) - The percentage of the CPU limit per model replica that is currently utilized by a specific model. Memory utilization (%) - The percentage of the memory limit per model replica that is utilized by a specific model. You can specify a time range and a refresh interval for these metrics to help you determine, for example, when the peak usage hours are and how the model is performing at a specified time. Prerequisites You have installed Red Hat OpenShift AI. You have logged in to Red Hat OpenShift AI. If you are using OpenShift AI groups, you are part of the user group or admin group (for example, rhoai-users or rhoai-admins ) in OpenShift. The following dashboard configuration options are set to the default values as shown: For more information, see Dashboard configuration options . You have deployed a model on the single-model serving platform by using a preinstalled runtime. Note Metrics are only supported for models deployed by using a preinstalled model-serving runtime or a custom runtime that is duplicated from a preinstalled runtime. Procedure From the OpenShift AI dashboard navigation menu, click Data Science Projects . The Data Science Projects page opens. Click the name of the project that contains the data science models that you want to monitor. In the project details page, click the Models tab. Select the model that you are interested in. On the Endpoint performance tab, set the following options: Time range - Specifies how long to track the metrics. You can select one of these values: 1 hour, 24 hours, 7 days, and 30 days. Refresh interval - Specifies how frequently the graphs on the metrics page are refreshed (to show the latest data). You can select one of these values: 15 seconds, 30 seconds, 1 minute, 5 minutes, 15 minutes, 30 minutes, 1 hour, 2 hours, and 1 day. Scroll down to view data graphs for number of requests, average response time, CPU utilization, and memory utilization. Verification The Endpoint performance tab shows graphs of metrics for the model. 2.15.2. Deploying a Grafana metrics dashboard You can deploy a Grafana metrics dashboard for User Workload Monitoring (UWM) to monitor performance and resource usage metrics for models deployed on the single-model serving platform. You can create a Kustomize overlay, similar to this example . Use the overlay to deploy preconfigured metrics dashboards for models deployed with OpenVino Model Server (OVMS) and vLLM. Prerequisites You have cluster admin privileges for your OpenShift cluster. You have installed the OpenShift command-line interface (CLI). For more information, see Installing the OpenShift CLI (OpenShift Dedicated) or Installing the OpenShift CLI (Red Hat OpenShift Service on AWS) . You have created an overlay to deploy a Grafana instance, similar to this example . Note To view GPU metrics, you must enable the NVIDIA GPU monitoring dashboard as described in Enabling the GPU monitoring dashboard . The GPU monitoring dashboard provides a comprehensive view of GPU utilization, memory usage, and other metrics for your GPU nodes. Procedure In a terminal window, log in to the OpenShift CLI as a cluster administrator. If you have not already created the overlay to install the Grafana operator and metrics dashboards, refer to the RHOAI UWM repository to create it. Install the Grafana instance and metrics dashboards on your OpenShift cluster with the overlay that you created. Replace <overlay-name> with the name of your overlay. Retrieve the URL of the Grafana instance. Replace <namespace> with the namespace that contains the Grafana instance. You see output similar to the following example. Use the URL to access the Grafana instance: Verification You can access the preconfigured dashboards available for KServe, vLLM and OVMS on the Grafana instance. 2.16. Optimizing model-serving runtimes You can optionally enhance the preinstalled model-serving runtimes available in OpenShift AI to leverage additional benefits and capabilities, such as optimized inferencing, reduced latency, and fine-tuned resource allocation. 2.16.1. Enabling speculative decoding and multi-modal inferencing You can configure the vLLM ServingRuntime for KServe runtime to use speculative decoding, a parallel processing technique to optimize inferencing time for large language models (LLMs). You can also configure the runtime to support inferencing for vision-language models (VLMs). VLMs are a subset of multi-modal models that integrate both visual and textual data. The following procedure describes customizing the vLLM ServingRuntime for KServe runtime for speculative decoding and multi-modal inferencing. Prerequisites You have logged in to OpenShift AI as a user with OpenShift AI administrator privileges. If you are using the vLLM model-serving runtime for speculative decoding with a draft model, you have stored the original model and the speculative model in the same folder within your S3-compatible object storage. Procedure Follow the steps to deploy a model as described in Depyoing models on the single-model serving platform . In the Serving runtime field, select the vLLM ServingRuntime for KServe runtime. To configure the vLLM model-serving runtime for speculative decoding by matching n-grams in the prompt, add the following arguments under Additional serving runtime arguments in the Configuration parameters section: Replace <NUM_SPECULATIVE_TOKENS> and <NGRAM_PROMPT_LOOKUP_MAX> with your own values. Note Inferencing throughput varies depending on the model used for speculating with n-grams. To configure the vLLM model-serving runtime for speculative decoding with a draft model, add the following arguments under Additional serving runtime arguments in the Configuration parameters section: Replace <path_to_speculative_model> and <path_to_original_model> with the paths to the speculative model and original model on your S3-compatible object storage. Replace <NUM_SPECULATIVE_TOKENS> with your own value. To configure the vLLM model-serving runtime for multi-modal inferencing, add the following arguments under Additional serving runtime arguments in the Configuration parameters section: Note Only use the --trust-remote-code argument with models from trusted sources. Click Deploy . Verification If you have configured the vLLM model-serving runtime for speculative decoding, use the following example command to verify API requests to your deployed model: If you have configured the vLLM model-serving runtime for multi-modal inferencing, use the following example command to verify API requests to the vision-language model (VLM) that you have deployed: Additional resources vLLM: Engine Arguments OpenAI Compatible Server 2.17. Performance optimization and tuning 2.17.1. Determining GPU requirements for LLM-powered applications There are several factors to consider when choosing GPUs for applications powered by a Large Language Model (LLM) hosted on OpenShift AI. The following guidelines help you determine the hardware requirements for your application, depending on the size and expected usage of your model. Estimating memory needs : A general rule of thumb is that a model with N parameters in 16-bit precision requires approximately 2N bytes of GPU memory. For example, an 8-billion-parameter model requires around 16GB of GPU memory, while a 70-billion-parameter model requires around 140GB. Quantization : To reduce memory requirements and potentially improve throughput, you can use quantization to load or run the model at lower-precision formats such as INT8, FP8, or INT4. This reduces the memory footprint at the expense of a slight reduction in model accuracy. Note The vLLM ServingRuntime for KServe model-serving runtime supports several quantization methods. For more information about supported implementations and compatible hardware, see Supported hardware for quantization kernels . Additional memory for key-value cache : In addition to model weights, GPU memory is also needed to store the attention key-value (KV) cache, which increases with the number of requests and the sequence length of each request. This can impact performance in real-time applications, especially for larger models. Recommended GPU configurations : Small Models (1B-8B parameters) : For models in the range, a GPU with 24GB of memory is generally sufficient to support a small number of concurrent users. Medium Models (10B-34B parameters) : Models under 20B parameters require at least 48GB of GPU memory. Models that are between 20B - 34B parameters require at least 80GB or more of memory in a single GPU. Large Models (70B parameters) : Models in this range may need to be distributed across multiple GPUs by using tensor parallelism techniques. Tensor parallelism allows the model to span multiple GPUs, improving inter-token latency and increasing the maximum batch size by freeing up additional memory for KV cache. Tensor parallelism works best when GPUs have fast interconnects such as an NVLink. Very Large Models (405B parameters) : For extremely large models, quantization is recommended to reduce memory demands. You can also distribute the model using pipeline parallelism across multiple GPUs, or even across two servers. This approach allows you to scale beyond the memory limitations of a single server, but requires careful management of inter-server communication for optimal performance. For best results, start with smaller models and then scale up to larger models as required, using techniques such as parallelism and quantization to meet your performance and memory requirements. Additional resources Distributed serving 2.17.2. Performance considerations for text-summarization and retrieval-augmented generation (RAG) applications There are additional factors that need to be taken into consideration for text-summarization and RAG applications, as well as for LLM-powered services that process large documents uploaded by users. Longer Input Sequences : The input sequence length can be significantly longer than in a typical chat application, if each user query includes a large prompt or a large amount of context such as an uploaded document. The longer input sequence length increases the prefill time , the time the model takes to process the initial input sequence before generating a response, which can then lead to a higher Time-to-First-Token (TTFT) . A longer TTFT may impact the responsiveness of the application. Minimize this latency for optimal user experience. KV Cache Usage : Longer sequences require more GPU memory for the key-value (KV) cache . The KV cache stores intermediate attention data to improve model performance during generation. A high KV cache utilization per request requires a hardware setup with sufficient GPU memory. This is particularly crucial if multiple users are querying the model concurrently, as each request adds to the total memory load. Optimal Hardware Configuration : To maintain responsiveness and avoid memory bottlenecks, select a GPU configuration with sufficient memory. For instance, instead of running an 8B model on a single 24GB GPU, deploying it on a larger GPU (e.g., 48GB or 80GB) or across multiple GPUs can improve performance by providing more memory headroom for the KV cache and reducing inter-token latency. Multi-GPU setups with tensor parallelism can also help manage memory demands and improve efficiency for larger input sequences. In summary, to ensure optimal responsiveness and scalability for document-based applications, you must prioritize hardware with high GPU memory capacity and also consider multi-GPU configurations to handle the increased memory requirements of long input sequences and KV caching. 2.17.3. Inference performance metrics Latency , throughput and cost per million tokens are key metrics to consider when evaluating the response generation efficiency of a model during inferencing. These metrics provide a comprehensive view of a model's inference performance and can help balance speed, efficiency, and cost for different use cases. 2.17.3.1. Latency Latency is critical for interactive or real-time use cases, and is measured using the following metrics: Time-to-First-Token (TTFT) : The delay in milliseconds between the initial request and the generation of the first token. This metric is important for streaming responses. Inter-Token Latency (ITL) : The time taken in milliseconds to generate each subsequent token after the first, also relevant for streaming. Time-Per-Output-Token (TPOT) : For non-streaming requests, the average time taken in milliseconds to generate each token in an output sequence. 2.17.3.2. Throughput Throughput measures the overall efficiency of a model server and is expressed with the following metrics: Tokens per Second (TPS) : The total number of tokens generated per second across all active requests. Requests per Second (RPS) : The number of requests processed per second. RPS, like response time, is sensitive to sequence length. 2.17.3.3. Cost per million tokens Cost per Million Tokens measures the cost-effectiveness of a model's inference, indicating the expense incurred per million tokens generated. This metric helps to assess both the economic feasibility and scalability of deploying the model. 2.17.4. Resolving CUDA out-of-memory errors In certain cases, depending on the model and hardware accelerator used, the TGIS memory auto-tuning algorithm might underestimate the amount of GPU memory needed to process long sequences. This miscalculation can lead to Compute Unified Architecture (CUDA) out-of-memory (OOM) error responses from the model server. In such cases, you must update or add additional parameters in the TGIS model-serving runtime, as described in the following procedure. Prerequisites You have logged in to OpenShift AI as a user with OpenShift AI administrator privileges. Procedure From the OpenShift AI dashboard, click Settings > Serving runtimes . The Serving runtimes page opens and shows the model-serving runtimes that are already installed and enabled. Based on the runtime that you used to deploy your model, perform one of the following actions: If you used the pre-installed TGIS Standalone ServingRuntime for KServe runtime, duplicate the runtime to create a custom version and then follow the remainder of this procedure. For more information about duplicating the pre-installed TGIS runtime, see Adding a custom model-serving runtime for the single-model serving platform . If you were already using a custom TGIS runtime, click the action menu (...) to the runtime and select Edit . The embedded YAML editor opens and shows the contents of the custom model-serving runtime. Add or update the BATCH_SAFETY_MARGIN environment variable and set the value to 30. Similarly, add or update the ESTIMATE_MEMORY_BATCH_SIZE environment variable and set the value to 8. Note The BATCH_SAFETY_MARGIN parameter sets a percentage of free GPU memory to hold back as a safety margin to avoid OOM conditions. The default value of BATCH_SAFETY_MARGIN is 20 . The ESTIMATE_MEMORY_BATCH_SIZE parameter sets the batch size used in the memory auto-tuning algorithm. The default value of ESTIMATE_MEMORY_BATCH_SIZE is 16 . Click Update . The Serving runtimes page opens and shows the list of runtimes that are installed. Observe that the custom model-serving runtime you updated is shown. To redeploy the model for the parameter updates to take effect, perform the following actions: From the OpenShift AI dashboard, click Model Serving > Deployed Models . Find the model you want to redeploy, click the action menu (...) to the model, and select Delete . Redeploy the model as described in Deploying models on the single-model serving platform . Verification You receive successful responses from the model server and no longer see CUDA OOM errors. 2.18. About the NVIDIA NIM model serving platform You can deploy models using NVIDIA NIM inference services on the NVIDIA NIM model serving platform . NVIDIA NIM, part of NVIDIA AI Enterprise, is a set of microservices designed for secure, reliable deployment of high performance AI model inferencing across clouds, data centers and workstations. Additional resources NVIDIA NIM 2.18.1. Enabling the NVIDIA NIM model serving platform As an administrator, you can use the Red Hat OpenShift AI dashboard to enable the NVIDIA NIM model serving platform. Prerequisites You have logged in to Red Hat OpenShift AI as an administrator. You have enabled the single-model serving platform. You do not need to enable a preinstalled runtime. For more information about enabling the single-model serving platform, see Enabling the single-model serving platform . The disableNIMModelServing OpenShift AI dashboard configuration is set to false : For more information, see Dashboard configuration options . You have enabled GPU support in OpenShift AI. This includes installing the Node Feature Discovery operator and NVIDIA GPU Operators. For more information, see Installing the Node Feature Discovery operator and Enabling NVIDIA GPUs . You have an NVIDIA Cloud Account (NCA) and can access the NVIDIA GPU Cloud (NGC) portal. For more information, see NVIDIA GPU Cloud user guide . Your NCA account is associated with the NVIDIA AI Enterprise Viewer role. You have generated an NGC API key on the NGC portal. For more information, see NGC API keys . Procedure Log in to OpenShift AI. In the left menu of the OpenShift AI dashboard, click Applications Explore . On the Explore page, find the NVIDIA NIM tile. Click Enable on the application tile. Enter the NGC API key and then click Submit . Verification The NVIDIA NIM application that you enabled appears on the Enabled page. 2.18.2. Deploying models on the NVIDIA NIM model serving platform When you have enabled the NVIDIA NIM model serving platform , you can start to deploy NVIDIA-optimized models on the platform. Prerequisites You have logged in to Red Hat OpenShift AI. If you are using OpenShift AI groups, you are part of the user group or admin group (for example, rhoai-users or rhoai-admins ) in OpenShift. You have enabled the NVIDIA NIM model serving platform . You have created a data science project. You have enabled support for graphic processing units (GPUs) in OpenShift AI. This includes installing the Node Feature Discovery operator and NVIDIA GPU Operators. For more information, see Installing the Node Feature Discovery operator and Enabling NVIDIA GPUs . Procedure In the left menu, click Data Science Projects . The Data Science Projects page opens. Click the name of the project that you want to deploy a model in. A project details page opens. Click the Models tab. In the Models section, perform one of the following actions: On the NVIDIA NIM model serving platform tile, click Select NVIDIA NIM on the tile, and then click Deploy model . If you have previously selected the NVIDIA NIM model serving type, the Models page displays NVIDIA model serving enabled on the upper-right corner, along with the Deploy model button. To proceed, click Deploy model . The Deploy model dialog opens. Configure properties for deploying your model as follows: In the Model deployment name field, enter a unique name for the deployment. From the NVIDIA NIM list, select the NVIDIA NIM model that you want to deploy. For more information, see Supported Models In the NVIDIA NIM storage size field, specify the size of the cluster storage instance that will be created to store the NVIDIA NIM model. In the Number of model server replicas to deploy field, specify a value. From the Model server size list, select a value. From the Accelerator list, select an accelerator. The Number of accelerators field appears. In the Number of accelerators field, specify the number of accelerators to use. The default value is 1. Optional: In the Model route section, select the Make deployed models available through an external route checkbox to make your deployed models available to external clients. To require token authentication for inference requests to the deployed model, perform the following actions: Select Require token authentication . In the Service account name field, enter the service account name that the token will be generated for. To add an additional service account, click Add a service account and enter another service account name. Click Deploy . Verification Confirm that the deployed model is shown on the Models tab for the project, and on the Model Serving page of the dashboard with a checkmark in the Status column. Additional resources NVIDIA NIM API reference Supported Models 2.18.3. Enabling NVIDIA NIM metrics for an existing NIM deployment If you have previously deployed a NIM model in OpenShift AI, and then upgraded to the latest version, you must manually enable NIM metrics for your existing deployment by adding annotations to enable metrics collection and graph generation. Note NIM metrics and graphs are automatically enabled for new deployments in the latest version of OpenShift AI. 2.18.3.1. Enabling graph generation for an existing NIM deployment The following procedure describes how to enable graph generation for an existing NIM deployment. Prerequisites You have cluster administrator privileges for your OpenShift cluster. You have downloaded and installed the OpenShift command-line interface (CLI). For more information, see Installing the OpenShift CLI (Red Hat OpenShift Dedicated) or Installing the OpenShift CLI (Red Hat OpenShift Service on AWS). You have an existing NIM deployment in OpenShift AI. Procedure In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI. Confirm the name of the ServingRuntime associated with your NIM deployment: Replace <namespace> with the namespace of the project where your NIM model is deployed. Check for an existing metadata.annotations section in the ServingRuntime configuration: Replace <servingruntime-name> with the name of the ServingRuntime from the step. Perform one of the following actions: If the metadata.annotations section is not present in the configuration, add the section with the required annotations: You see output similar to the following: If there is an existing metadata.annotations section, add the required annotations to the section: You see output similar to the following: Verification Confirm that the annotation has been added to the ServingRuntime of your existing NIM deployment. The annotation that you added appears in the output: Note For metrics to be available for graph generation, you must also enable metrics collection for your deployment. Please see Enabling metrics collection for an existing NIM deployment . 2.18.3.2. Enabling metrics collection for an existing NIM deployment To enable metrics collection for your existing NIM deployment, you must manually add the Prometheus endpoint and port annotations to the InferenceService of your deployment. The following procedure describes how to add the required Prometheus annotations to the InferenceService of your NIM deployment. Prerequisites You have cluster administrator privileges for your OpenShift cluster. You have downloaded and installed the OpenShift command-line interface (CLI). For more information, see Installing the OpenShift CLI (Red Hat OpenShift Dedicated) or Installing the OpenShift CLI (Red Hat OpenShift Service on AWS). You have an existing NIM deployment in OpenShift AI. Procedure In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI. Confirm the name of the InferenceService associated with your NIM deployment: Replace <namespace> with the namespace of the project where your NIM model is deployed. Check if there is an existing spec.predictor.annotations section in the InferenceService configuration: Replace <inferenceservice-name> with the name of the InferenceService from the step. Perform one of the following actions: If the spec.predictor.annotations section does not exist in the configuration, add the section and required annotations: The annotation that you added appears in the output: If there is an existing spec.predictor.annotations section, add the Prometheus annotations to the section: The annotations that you added appears in the output: Verification Confirm that the annotations have been added to the InferenceService . You see the annotation that you added in the output: 2.18.4. Viewing NVIDIA NIM metrics for a NIM model In OpenShift AI, you can observe the following NVIDIA NIM metrics for a NIM model deployed on the NVIDIA NIM model serving platform: GPU cache usage over time (ms) Current running, waiting, and max requests count Tokens count Time to first token Time per output token Request outcomes You can specify a time range and a refresh interval for these metrics to help you determine, for example, the peak usage hours and model performance at a specified time. Prerequisites You have enabled the NVIDIA NIM model serving platform. You have deployed a NIM model on the NVIDIA NIM model serving platform. If you are using OpenShift AI groups, you are part of the user group or admin group (for example, rhoai-users or rhoai-admins ) in OpenShift. The disableKServeMetrics OpenShift AI dashboard configuration option is set to its default value of false : For more information, see Dashboard configuration options . Procedure From the OpenShift AI dashboard navigation menu, click Data Science Projects . The Data Science Projects page opens. Click the name of the project that contains the NIM model that you want to monitor. In the project details page, click the Models tab. Click the NIM model that you want to observe. On the NIM Metrics tab, set the following options: Time range - Specifies how long to track the metrics. You can select one of these values: 1 hour, 24 hours, 7 days, and 30 days. Refresh interval - Specifies how frequently the graphs on the metrics page are refreshed (to show the latest data). You can select one of these values: 15 seconds, 30 seconds, 1 minute, 5 minutes, 15 minutes, 30 minutes, 1 hour, 2 hours, and 1 day. Scroll down to view data graphs for NIM metrics. Verification The NIM Metrics tab shows graphs of NIM metrics for the deployed NIM model. Additional resources NVIDIA NIM observability 2.18.5. Viewing performance metrics for a NIM model You can observe the following performance metrics for a NIM model deployed on the NVIDIA NIM model serving platform: Number of requests - The number of requests that have failed or succeeded for a specific model. Average response time (ms) - The average time it takes a specific model to respond to requests. CPU utilization (%) - The percentage of the CPU limit per model replica that is currently utilized by a specific model. Memory utilization (%) - The percentage of the memory limit per model replica that is utilized by a specific model. You can specify a time range and a refresh interval for these metrics to help you determine, for example, the peak usage hours and model performance at a specified time. Prerequisites You have enabled the NVIDIA NIM model serving platform. You have deployed a NIM model on the NVIDIA NIM model serving platform. If you are using OpenShift AI groups, you are part of the user group or admin group (for example, rhoai-users or rhoai-admins ) in OpenShift. The disableKServeMetrics OpenShift AI dashboard configuration option is set to its default value of false : For more information, see Dashboard configuration options . Procedure From the OpenShift AI dashboard navigation menu, click Data Science Projects . The Data Science Projects page opens. Click the name of the project that contains the NIM model that you want to monitor. In the project details page, click the Models tab. Click the NIM model that you want to observe. On the Endpoint performance tab, set the following options: Time range - Specifies how long to track the metrics. You can select one of these values: 1 hour, 24 hours, 7 days, and 30 days. Refresh interval - Specifies how frequently the graphs on the metrics page are refreshed to show the latest data. You can select one of these values: 15 seconds, 30 seconds, 1 minute, 5 minutes, 15 minutes, 30 minutes, 1 hour, 2 hours, and 1 day. Scroll down to view data graphs for performance metrics. Verification The Endpoint performance tab shows graphs of performance metrics for the deployed NIM model.

Commands:
apiVersion: serving.kserve.io/v1alpha1 kind: ServingRuntime metadata: annotations: opendatahub.io/recommended-accelerators: '["nvidia.com/gpu"]' 1 openshift.io/display-name: vLLM ServingRuntime for KServe 2 labels: opendatahub.io/dashboard: "true" name: vllm-runtime spec: annotations: prometheus.io/path: /metrics 3 prometheus.io/port: "8080" 4 containers : - args: - --port=8080 - --model=/mnt/models 5 - --served-model-name={{.Name}} 6 command: 7 - python - '-m' - vllm.entrypoints.openai.api_server env: - name: HF_HOME value: /tmp/hf_home image: 8 quay.io/modh/vllm@sha256:8a3dd8ad6e15fe7b8e5e471037519719d4d8ad3db9d69389f2beded36a6f5b21 name: kserve-container ports: - containerPort: 8080 protocol: TCP multiModel: false 9 supportedModelFormats: 10 - autoSelect: true name: vLLM
apiVersion: serving.kserve.io/v1beta1 kind: InferenceService metadata: annotations: openshift.io/display-name: granite serving.knative.openshift.io/enablePassthrough: 'true' sidecar.istio.io/inject: 'true' sidecar.istio.io/rewriteAppHTTPProbers: 'true' name: granite labels: opendatahub.io/dashboard: 'true' spec: predictor: maxReplicas: 1 minReplicas: 1 model: modelFormat: name: vLLM name: '' resources: limits: cpu: '6' memory: 24Gi nvidia.com/gpu: '1' requests: cpu: '1' memory: 8Gi nvidia.com/gpu: '1' runtime: vLLM ServingRuntime for KServe storage: key: aws-connection-my-storage path: models/granite-7b-instruct/ tolerations: - effect: NoSchedule key: nvidia.com/gpu operator: Exists
curl --json '{"model_id": "<model_name__>", "inputs": "<text>"}' https://<inference_endpoint_url>:443/api/v1/task/server-streaming-text-generation -H 'Authorization: Bearer <token>'
curl -H 'Content-Type: application/json' -d '{"inputs": "<text>", "model_id": "<model_id>"}' <inference_endpoint_url>/api/v1/task/embedding -H 'Authorization: Bearer <token>'
grpcurl -d '{"text": "<text>"}' -H \"mm-model-id: <model_id>\" <inference_endpoint_url>:443 caikit.runtime.Nlp.NlpService/EmbeddingTaskPredict -H 'Authorization: Bearer <token>'
grpcurl -proto text-generation-inference/proto/generation.proto -d '{"requests": [{"text":"<text>"}]}' -H 'Authorization: Bearer <token>' -insecure <inference_endpoint_url>:443 fmaas.GenerationService/Generate
curl -ks <inference_endpoint_url>/v2/models/<model_name>/infer -d '{ "model_name": "<model_name>", "inputs": [{ "name": "<name_of_model_input>", "shape": [<shape>], "datatype": "<data_type>", "data": [<data>] }]}' -H 'Authorization: Bearer <token>'
containers: - args: - --chat-template=<CHAT_TEMPLATE>
curl -v https://<inference_endpoint_url>:443/v1/chat/completions -H "Content-Type: application/json" -d '{ "messages": [{ "role": "<role>", "content": "<content>" }] -H 'Authorization: Bearer <token>'
curl -ks <inference_endpoint_url>/v2/models/<model_name>/infer -d '{ "model_name": "<model_name>", "inputs": [{ "name": "<name_of_model_input>", "shape": [<shape>], "datatype": "<data_type>", "data": [<data>] }]}' -H 'Authorization: Bearer <token>'
grpcurl -cacert ./openshift_ca_istio_knative.crt -proto ./grpc_predict_v2.proto -d @ -H "Authorization: Bearer <token>" <inference_endpoint_url>:443 inference.GRPCInferenceService/ModelMetadata
apiVersion: serving.kserve.io/v1alpha1 kind: ServingRuntime metadata: name: triton-kserve-rest labels: opendatahub.io/dashboard: "true" spec: annotations: prometheus.kserve.io/path: /metrics prometheus.kserve.io/port: "8002" containers: - args: - tritonserver - --model-store=/mnt/models - --grpc-port=9000 - --http-port=8080 - --allow-grpc=true - --allow-http=true image: nvcr.io/nvidia/tritonserver@sha256:xxxxx name: kserve-container resources: limits: cpu: "1" memory: 2Gi requests: cpu: "1" memory: 2Gi ports: - containerPort: 8080 protocol: TCP protocolVersions: - v2 - grpc-v2 supportedModelFormats: - autoSelect: true name: tensorrt version: "8" - autoSelect: true name: tensorflow version: "1" - autoSelect: true name: tensorflow version: "2" - autoSelect: true name: onnx version: "1" - name: pytorch version: "1" - autoSelect: true name: triton version: "2" - autoSelect: true name: xgboost version: "1" - autoSelect: true name: python version: "1"
apiVersion: serving.kserve.io/v1alpha1 kind: ServingRuntime metadata: name: triton-kserve-grpc labels: opendatahub.io/dashboard: "true" spec: annotations: prometheus.kserve.io/path: /metrics prometheus.kserve.io/port: "8002" containers: - args: - tritonserver - --model-store=/mnt/models - --grpc-port=9000 - --http-port=8080 - --allow-grpc=true - --allow-http=true image: nvcr.io/nvidia/tritonserver@sha256:xxxxx name: kserve-container ports: - containerPort: 9000 name: h2c protocol: TCP volumeMounts: - mountPath: /dev/shm name: shm resources: limits: cpu: "1" memory: 2Gi requests: cpu: "1" memory: 2Gi protocolVersions: - v2 - grpc-v2 supportedModelFormats: - autoSelect: true name: tensorrt version: "8" - autoSelect: true name: tensorflow version: "1" - autoSelect: true name: tensorflow version: "2" - autoSelect: true name: onnx version: "1" - name: pytorch version: "1" - autoSelect: true name: triton version: "2" - autoSelect: true name: xgboost version: "1" - autoSelect: true name: python version: "1" volumes: - emptyDir: null medium: Memory sizeLimit: 2Gi name: shm
apiVersion: serving.kserve.io/v1alpha1 kind: ServingRuntime metadata: name: kserve-triton annotations: openshift.io/display-name: Triton ServingRuntime
apiVersion: serving.kserve.io/v1alpha1 kind: InferenceService metadata: name: my-inference-service spec: predictor: annotations: serving.knative.dev/progress-deadline: 30m
get -o json inferenceservice <inferenceservicename/modelname> -n <projectname>
cd USD(mktemp -d)
mkdir -p models/1
DOWNLOAD_URL=https://github.com/onnx/models/raw/main/validated/vision/classification/mobilenet/model/mobilenetv2-7.onnx curl -L USDDOWNLOAD_URL -O --output-dir models/1/
tree
. ├── Containerfile └── models └── 1 └── mobilenetv2-7.onnx
FROM registry.access.redhat.com/ubi9/ubi-micro:latest COPY --chown=0:0 models /models RUN chmod -R a=rX /models nobody user USER 65534
build --format=oci -t quay.io/<user_name>/<repository_name>:<tag_name> . push quay.io/<user_name>/<repository_name>:<tag_name>
new-project oci-model-example
process -n redhat-ods-applications -o yaml kserve-ovms | oc apply -f -
get servingruntimes
NAME DISABLED MODELTYPE CONTAINERS AGE kserve-ovms openvino_ir kserve-container 1m
apiVersion: serving.kserve.io/v1beta1 kind: InferenceService metadata: name: sample-isvc-using-oci spec: predictor: model: runtime: kserve-ovms # Ensure this matches the name of the ServingRuntime resource modelFormat: name: onnx storageUri: oci://quay.io/<user_name>/<repository_name>:<tag_name> resources: requests: memory: 500Mi cpu: 100m # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it limits: memory: 4Gi cpu: 500m # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it
apiVersion: serving.kserve.io/v1beta1 kind: InferenceService metadata: name: sample-isvc-using-private-oci spec: predictor: model: runtime: kserve-ovms # Ensure this matches the name of the ServingRuntime resource modelFormat: name: onnx storageUri: oci://quay.io/<user_name>/<repository_name>:<tag_name> resources: requests: memory: 500Mi cpu: 100m # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it limits: memory: 4Gi cpu: 500m # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it imagePullSecrets: # Specify image pull secrets to use for fetching container images, including OCI model images - name: <pull-secret-name>
get inferenceservice
`VLLM_SKIP_WARMUP="true"`
--distributed-executor-backend=mp 1 --max-model-len=6144 2
--distributed-executor-backend=mp 1
--distributed-executor-backend=mp 1 --tensor-parallel-size=4 2 --max-model-len=6448 3
--distributed-executor-backend=mp 1 --max-model-len=15344 2
curl -q -X 'POST' "https://<inference_endpoint_url>:443/v1/chat/completions" -H 'accept: application/json' -H 'Content-Type: application/json' -d "{ \"model\": \"<model_name>\", \"prompt\": \"<prompt>", \"max_tokens\": <max_tokens>, \"temperature\": <temperature> }"
oc login <openshift_cluster_url> -u <admin_username> -p <password>
new-project kserve-demo
apply -f - apiVersion: v1 kind: PersistentVolumeClaim metadata: name: granite-8b-code-base-pvc spec: accessModes: - ReadWriteMany volumeMode: Filesystem resources: requests: storage: <model size> storageClassName: <storage class>
apiVersion: v1 kind: Pod metadata: name: download-granite-8b-code labels: name: download-granite-8b-code spec: volumes: - name: model-volume persistentVolumeClaim: claimName: granite-8b-code-claim restartPolicy: Never initContainers: - name: fix-volume-permissions image: quay.io/quay/busybox@sha256:92f3298bf80a1ba949140d77987f5de081f010337880cd771f7e7fc928f8c74d command: ["sh"] args: ["-c", "mkdir -p /mnt/models/USD(MODEL_PATH) && chmod -R 777 /mnt/models"] 1 volumeMounts: - mountPath: "/mnt/models/" name: model-volume env: - name: MODEL_PATH value: <model path> 2 containers: - resources: requests: memory: 40Gi name: download-model imagePullPolicy: IfNotPresent image: quay.io/opendatahub/kserve-storage-initializer:v0.14 3 args: - 's3://USD(BUCKET_NAME)/USD(MODEL_PATH)/' - /mnt/models/USD(MODEL_PATH) env: - name: AWS_ACCESS_KEY_ID value: <id> 4 - name: AWS_SECRET_ACCESS_KEY value: <secret> 5 - name: BUCKET_NAME value: <bucket_name> 6 - name: MODEL_PATH value: <model path> 7 - name: S3_USE_HTTPS value: "1" - name: AWS_ENDPOINT_URL value: <AWS endpoint> 8 - name: awsAnonymousCredential value: 'false' - name: AWS_DEFAULT_REGION value: <region> 9 - name: S3_VERIFY_SSL value: 'true' 10 volumeMounts: - mountPath: "/mnt/models/" name: model-volume
process vllm-multinode-runtime-template -n redhat-ods-applications|oc apply -n kserve-demo -f -
apiVersion: serving.kserve.io/v1beta1 kind: InferenceService metadata: annotations: serving.kserve.io/deploymentMode: RawDeployment serving.kserve.io/autoscalerClass: external name: <inference service name> spec: predictor: model: modelFormat: name: vLLM runtime: vllm-multinode-runtime storageUri: pvc://<pvc name>/<model path> workerSpec: {}
Get pod name podName=USD(oc get pod -l app=isvc.granite-8b-code-base-pvc-predictor --no-headers|cut -d' ' -f1) workerPodName=USD(oc get pod -l app=isvc.granite-8b-code-base-pvc-predictor-worker --no-headers|cut -d' ' -f1) wait --for=condition=ready pod/USD{podName} --timeout=300s Check the GPU memory size for both the head and worker pods: echo "### HEAD NODE GPU Memory Size" exec USDpodName -- nvidia-smi echo "### Worker NODE GPU Memory Size" exec USDworkerPodName -- nvidia-smi
+-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.90.07 Driver Version: 550.90.07 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | | 0% 33C P0 71W / 300W |19031MiB / 23028MiB <1>| 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.90.07 Driver Version: 550.90.07 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | | 0% 30C P0 69W / 300W |18959MiB / 23028MiB <2>| 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+
wait --for=condition=ready pod/USD{podName} -n USDDEMO_NAMESPACE --timeout=300s export MODEL_NAME=granite-8b-code-base-pvc
NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE granite-8b-code-base-pvc http://granite-8b-code-base-pvc.default.example.com
wait --for=condition=ready pod/USD{podName} -n vllm-multinode --timeout=300s port-forward USDpodName 8080:8080 & curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d "{ 'model': "USDMODEL_NAME", 'prompt': 'At what temperature does Nitrogen boil?', 'max_tokens': 100, 'temperature': 0 }"
sum(increase(vllm:request_success_total{namespace= USD{namespace} ,model_name= USD{model_name} }[USD{rate_interval}]))
sum(increase(tgi_request_success{namespace=USD{namespace}, pod=~ USD{model_name}-predictor-.* }[USD{rate_interval}]))
sum(increase(predict_rpc_count_total{namespace= USD{namespace} ,code= OK ,model_id= USD{model_name} }[USD{rate_interval}]))
sum(increase(ovms_requests_success{namespace= USD{namespace} ,name= USD{model_name} }[USD{rate_interval}]))
disablePerformanceMetrics:false disableKServeMetrics:false
apply -k overlays/<overlay-name>
get route -n <namespace> grafana-route -o jsonpath='{.spec.host}'
grafana-<namespace>.apps.example-openshift.com
--speculative-model=[ngram] --num-speculative-tokens=<NUM_SPECULATIVE_TOKENS> --ngram-prompt-lookup-max=<NGRAM_PROMPT_LOOKUP_MAX> --use-v2-block-manager
--port=8080 --served-model-name={{.Name}} --distributed-executor-backend=mp --model=/mnt/models/<path_to_original_model> --speculative-model=/mnt/models/<path_to_speculative_model> --num-speculative-tokens=<NUM_SPECULATIVE_TOKENS> --use-v2-block-manager
--trust-remote-code
curl -v https://<inference_endpoint_url>:443/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer <token>"
curl -v https://<inference_endpoint_url>:443/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer <token>" -d '{"model":"<model_name>", "messages": [{"role":"<role>", "content": [{"type":"text", "text":"<text>" }, {"type":"image_url", "image_url":"<image_url_link>" } ] } ] }'
spec: containers: env: - name: BATCH_SAFETY_MARGIN value: 30 - name: ESTIMATE_MEMORY_BATCH value: 8
disableNIMModelServing: false
get servingruntime -n <namespace>
get servingruntime -n <namespace> <servingruntime-name> -o json | jq '.metadata.annotations'
patch servingruntime -n <namespace> <servingruntime-name> --type json --patch '[{"op": "add", "path": "/metadata/annotations", "value": {"runtimes.opendatahub.io/nvidia-nim": "true"}}]'
servingruntime.serving.kserve.io/nim-serving-runtime patched
patch servingruntime -n <project-namespace> <runtime-name> --type json --patch '[{"op": "add", "path": "/metadata/annotations/runtimes.opendatahub.io~1nvidia-nim", "value": "true"}]'
servingruntime.serving.kserve.io/nim-serving-runtime patched
get servingruntime -n <namespace> <servingruntime-name> -o json | jq '.metadata.annotations'
"runtimes.opendatahub.io/nvidia-nim": "true"
get inferenceservice -n <namespace>
get inferenceservice -n <namespace> <inferenceservice-name> -o json | jq '.spec.predictor.annotations'
patch inferenceservice -n <namespace> <inference-name> --type json --patch '[{"op": "add", "path": "/spec/predictor/annotations", "value": {"prometheus.io/path": "/metrics", "prometheus.io/port": "8000"}}]'
inferenceservice.serving.kserve.io/nim-serving-runtime patched
patch inferenceservice -n <namespace> <inference-service-name> --type json --patch '[{"op": "add", "path": "/spec/predictor/annotations/prometheus.io~1path", "value": "/metrics"}, {"op": "add", "path": "/spec/predictor/annotations/prometheus.io~1port", "value": "8000"}]'
inferenceservice.serving.kserve.io/nim-serving-runtime patched
get inferenceservice -n <namespace> <inferenceservice-name> -o json | jq '.spec.predictor.annotations'
prometheus.io/path: /metrics
prometheus.io/port: 8000
disableKServeMetrics: false
disableKServeMetrics: false