Chapter 7. Configuring the Guardrails Orchestrator service Important The Guardrails Orchestrator service is currently available in Red Hat OpenShift AI 2.18 as a Technology Preview feature. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . The TrustyAI Guardrails Orchestrator service is a tool to invoke detections on text generation inputs and outputs, as well as standalone detections. It is underpinned by the open-source project FMS-Guardrails Orchestrator from IBM. You can deploy the Guardrails Orchestrator service through a Custom Resource Definition (CRD) that is managed by the TrustyAI Operator. The following sections describe how to do the following tasks: Set up the Guardrails Orchestrator service Create a custom resource (CR) Deploy a Guardrails Orchestrator instance Monitor user-inputs to your LLM using this service 7.1. Deploying the Guardrails Orchestrator service You can deploy a Guardrails Orchestrator instance in your namespace to monitor elements, such as user inputs to your Large Language Model (LLM). Prerequisites You have cluster administrator privileges for your OpenShift cluster. You have downloaded and installed the OpenShift command-line interface (CLI). See Installing the OpenShift CLI . You are familiar with creating a config map for monitoring a user-defined workflow. You perform similar steps in this procedure. You have KServe set to RawDeployment . See Deploying models on single-node OpenShift using KServe Raw Deployment mode . You have the TrustyAI component in your OpenShift AI DataScienceCluster set to Managed . You have an LLM for chat generation deployed in your namespace. You have an LLM for text classification deployed in your namespace. Procedure Define a ConfigMap object in a YAML file to specify the chat_generation and detectors services. For example, create a file named orchestrator_cm.yaml with the following content: Example orchestrator_cm.yaml --- kind: ConfigMap apiVersion: v1 metadata: name: fms-orchestr8-config-nlp data: config.yaml: | chat_generation: 1 service: hostname: <CHAT_GENERATION_HOSTNAME> port: 8080 detectors: 2 <DETECTOR_NAME>: type: text_contents service: hostname: <DETECTOR_HOSTNAME> port: 8000 chunker_id: whole_doc_chunker default_threshold: 0.5 --- <1> A service for chat generation referring to a deployed LLM in your namespace where you are adding guardrails. <2> A list of services responsible for running detection of a certain class of content on text spans. Each of these services refer to a deployed LLM for text classification in your namespace. Deploy the orchestrator_cm.yaml config map: --- USD oc apply -f orchestrator_cm.yaml -n <TEST_NAMESPACE> --- Specify the previously created ConfigMap object created in the GuardrailsOrchestrator custom resource (CR). For example, create a file named orchestrator_cr.yaml with the following content: Example orchestrator_cr.yaml CR --- apiVersion: trustyai.opendatahub.io/v1alpha1 kind: GuardrailsOrchestrator metadata: name: gorch-sample spec: orchestratorConfig: "fms-orchestr8-config-nlp" replicas: 1 --- Deploy the orchestrator CR, which creates a service account, deployment, service, and route object in your namespace. --- oc apply -f orchestrator_cr.yaml -n <TEST_NAMESPACE> --- Verification Confirm that the orchestrator and LLM pods are running: --- USD oc get pods -n <TEST_NAMESPACE> --- Example response --- NAME READY STATUS RESTARTS AGE gorch-test-55bf5f84d9-dd4vm 3/3 Running 0 3h53m ibm-container-deployment-bd4d9d898-52r5j 1/1 Running 0 3h53m ibm-hap-predictor-5d54c877d5-rbdms 1/1 Running 0 3h53m llm-container-deployment-bd4d9d898-52r5j 1/1 Running 0 3h53m llm-predictor-5d54c877d5-rbdms 1/1 Running 0 57m --- Query the /health endpoint of the orchestrator route to check the current status of the detector and generator services. If a 200 OK response is returned, the services are functioning normally: --- USD GORCH_ROUTE_HEALTH=USD(oc get routes gorch-test-health -o jsonpath='{.spec.host}') --- --- USD curl -v https://USDGORCH_ROUTE_HEALTH/health --- Example response --- * Trying ::1:8034... * connect to ::1 port 8034 failed: Connection refused * Trying 127.0.0.1:8034... * Connected to localhost (127.0.0.1) port 8034 (#0) > GET /health HTTP/1.1 > Host: localhost:8034 > User-Agent: curl/7.76.1 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < content-type: application/json < content-length: 36 < date: Fri, 31 Jan 2025 14:04:25 GMT < * Connection #0 to host localhost left intact {"fms-guardrails-orchestr8":"0.1.0"} --- 7.2. Guardrails Orchestrator parameters A GuardrailsOrchestrator object represents an orchestration service that invokes detectors on text generation input and output and standalone detections. You can modify the following parameters for the GuardrailsOrchestrator object you created previously: Parameter Description replicas The number of orchestrator pods to activate orchestratorConfig The name of the ConfigMap object that contains generator, detector, and chunker arguments. otelExporter **(optional)** A list of paired name and value arguments for configuring OpenTelemetry traces or metrics, or both: protocol - Sets the protocol for all the OpenTelemetry protocol (OTLP) endpoints. Valid values are grpc or http tracesProtocol - Sets the protocol for traces. Acceptable values are grpc or http metricsProtocol - Sets the protocol for metrics. Acceptable values are grpc or http otlpEndpoint - Sets the OTLP endpoint. Default values are gRPC localhost:4317 and HTTP localhost:4318 metricsEndpoint - Sets the OTLP endpoint for metrics tracesEndpoint - Sets the OTLP endpoint for traces 7.3. Configuring the OpenTelemetry Exporter for metrics and tracing Enable traces and metrics that are provided for the observability of the GuardrailsOrchestrator service with the OpenTelemetry Operator. Prerequisites You have installed the Red Hat OpenShift AI distributed tracing platform from the OperatorHub and created a Jaeger instance using the default settings. You have installed the Red Hat build of OpenTelemetry from the OperatorHub and created an OpenTelemetry instance. Procedure Define a GuardrailsOrchestrator custom resource object to specify the otelExporter configurations in a YAML file named orchestrator_otel_cr.yaml : Example of a orchestrator_otel_cr.yaml object that has OpenTelemetry configured: + --- apiVersion: trustyai.opendatahub.io/v1alpha1 kind: GuardrailsOrchestrator metadata: name: gorch-test spec: orchestratorConfig: "fms-orchestr8-config-nlp" 1 vllmGatewayConfig: "fms-orchestr8-config-gateway" 2 replicas: 1 otelExporter: protocol: "http" otlpEndpoint: "localhost:4318" otlpExport: "metrics" --- <1> These speficications are the same as Step 7 from "Configuring the regex detector and vLLM gateway". This example CR adds `otelExporter` configurations. Deploy the orchestrator custom resource. --- USD oc apply -f orchestrator_otel_cr.yaml ---

Commands:
--- kind: ConfigMap apiVersion: v1 metadata: name: fms-orchestr8-config-nlp data: config.yaml: | chat_generation: 1 service: hostname: <CHAT_GENERATION_HOSTNAME> port: 8080 detectors: 2 <DETECTOR_NAME>: type: text_contents service: hostname: <DETECTOR_HOSTNAME> port: 8000 chunker_id: whole_doc_chunker default_threshold: 0.5 --- <1> A service for chat generation referring to a deployed LLM in your namespace where you are adding guardrails. <2> A list of services responsible for running detection of a certain class of content on text spans. Each of these services refer to a deployed LLM for text classification in your namespace.
--- oc apply -f orchestrator_cm.yaml -n <TEST_NAMESPACE> ---
--- apiVersion: trustyai.opendatahub.io/v1alpha1 kind: GuardrailsOrchestrator metadata: name: gorch-sample spec: orchestratorConfig: "fms-orchestr8-config-nlp" replicas: 1 ---
--- apply -f orchestrator_cr.yaml -n <TEST_NAMESPACE> ---
--- oc get pods -n <TEST_NAMESPACE> ---
--- NAME READY STATUS RESTARTS AGE gorch-test-55bf5f84d9-dd4vm 3/3 Running 0 3h53m ibm-container-deployment-bd4d9d898-52r5j 1/1 Running 0 3h53m ibm-hap-predictor-5d54c877d5-rbdms 1/1 Running 0 3h53m llm-container-deployment-bd4d9d898-52r5j 1/1 Running 0 3h53m llm-predictor-5d54c877d5-rbdms 1/1 Running 0 57m ---
--- GORCH_ROUTE_HEALTH=USD(oc get routes gorch-test-health -o jsonpath='{.spec.host}') ---
--- curl -v https://USDGORCH_ROUTE_HEALTH/health ---
--- * Trying ::1:8034 * connect to ::1 port 8034 failed: Connection refused * Trying 127.0.0.1:8034 * Connected to localhost (127.0.0.1) port 8034 (#0) > GET /health HTTP/1.1 > Host: localhost:8034 > User-Agent: curl/7.76.1 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < content-type: application/json < content-length: 36 < date: Fri, 31 Jan 2025 14:04:25 GMT < * Connection #0 to host localhost left intact {"fms-guardrails-orchestr8":"0.1.0"} ---
--- apiVersion: trustyai.opendatahub.io/v1alpha1 kind: GuardrailsOrchestrator metadata: name: gorch-test spec: orchestratorConfig: "fms-orchestr8-config-nlp" 1 vllmGatewayConfig: "fms-orchestr8-config-gateway" 2 replicas: 1 otelExporter: protocol: "http" otlpEndpoint: "localhost:4318" otlpExport: "metrics" --- <1> These speficications are the same as Step 7 from "Configuring the regex detector and vLLM gateway". This example CR adds `otelExporter` configurations.
--- oc apply -f orchestrator_otel_cr.yaml ---