Chapter 5. Monitoring data drift As a data scientist, you might want to monitor your deployed models for data drift. Data drift refers to changes in the distribution or properties of incoming data that differ significantly from the data on which the model was originally trained. Detecting data drift helps ensure that your models continue to perform as expected and that they remain accurate and reliable. You can use data drift monitoring metrics from TrustyAI in Red Hat OpenShift AI to provide a quantitative measure of the alignment between the training data and the inference data. For information about the specific data drift metrics, see Supported drift metrics . 5.1. Creating a drift metric To monitor a deployed model for data drift, you must first create drift metrics. For information about the specific data drift metrics, see Supported drift metrics . For the complete list of TrustyAI metrics, see TrustyAI service API . 5.1.1. Creating a drift metric by using the CLI You can use the OpenShift command-line interface (CLI) to create a data drift metric for a model. Prerequisites You are familiar with the specific data set schema and understand the relevant inputs and outputs. Your OpenShift cluster administrator added you as a user to the OpenShift cluster and has installed the TrustyAI service for the data science project that contains the deployed models. You set up TrustyAI for your data science project, as described in Setting up TrustyAI for your project . Procedure Open a new terminal window. Follow these steps to log in to your OpenShift cluster: In the upper-right corner of the OpenShift web console, click your user name and select Copy login command . After you have logged in, click Display token . Copy the Log in with this token command and paste it in the OpenShift command-line interface (CLI). Set the TRUSTY_ROUTE variable to the external route for the TrustyAI service pod. Optionally, get the full list of TrustyAI service endpoints and payloads. Use POST /metrics/drift/meanshift/request to schedule a recurring drift monitoring metric with the following syntax and payload structure: Syntax : Payload structure : modelId The name of the model to monitor. referenceTag The data to use as the reference distribution. For example: 5.2. Deleting a drift metric by using the CLI You can use the OpenShift command-line interface (CLI) to delete a drift metric for a model. Prerequisites You have installed the OpenShift CLI ( oc ). You have a user token for authentication as described in Authenticating the TrustyAI service . There is an existing drift metric that you want to delete. Procedure Open a new terminal window. Follow these steps to log in to your OpenShift cluster: In the OpenShift web console, click your user name and select Copy login command . After you have logged in, click Display token . Copy the Log in with this token command and paste it in the OpenShift command-line interface (CLI). In the OpenShift CLI, get the route to the TrustyAI service: Optional: To list all currently active requests for a metric, use GET /metrics/{{metric}}/requests . For example, to list all currently scheduled MeanShift metrics, type: Alternatively, to list all currently scheduled metric requests, use GET /metrics/all/requests . To delete a metric, send an HTTP DELETE request to the /metrics/USDMETRIC/request endpoint to stop the periodic calculation, including the id of periodic task that you want to cancel in the payload. For example: Verification Use GET /metrics/{{metric}}/requests to list all currently active requests for the metric and verify the metric that you deleted is not shown. For example: 5.3. Viewing data drift metrics for a model After you create data drift monitoring metrics, use the OpenShift web console to view and update the metrics that you configured. Prerequisites You have access to the OpenShift cluster as a developer or as a user with view permissions for the project that you are viewing metrics for. You are familiar with querying metrics in user-defined projects. See Monitoring project and application metrics using the Developer perspective in Red Hat OpenShift Dedicated or Monitoring project and application metrics using the Developer perspective in Red Hat OpenShift Service on AWS . Procedure Log in to the OpenShift web console. Switch to the Developer perspective. In the left menu, click Observe . As described in Monitoring your project metrics in Red Hat OpenShift Dedicated or Monitoring your project metrics in Red Hat OpenShift Service on AWS , use the web console to run queries for trustyai_* metrics. 5.4. Supported drift metrics Red Hat OpenShift AI supports the following data drift metrics: MeanShift The MeanShift metric calculates the per-column probability that the data values in a test data set are from the same distribution as those in a training data set (assuming that the values are normally distributed). This metric measures the difference in the means of specific features between the two datasets. MeanShift is useful for identifying straightforward changes in data distributions, such as when the entire distribution has shifted to the left or right along the feature axis. This metric returns the probability that the distribution seen in the "real world" data is derived from the same distribution as the reference data. The closer the value is to 0, the more likely there is to be significant drift. FourierMMD The FourierMMD metric provides the probability that the data values in a test data set have drifted from the training data set distribution, assuming that the computed Maximum Mean Discrepancy (MMD) values are normally distributed. This metric compares the empirical distributions of the data sets by using an MMD measure in the Fourier domain. FourierMMD is useful for detecting subtle shifts in data distributions that might be overlooked by simpler statistical measures. This metric returns the probability that the distribution seen in the "real world" data has drifted from the reference data. The closer the value is to 1, the more likely there is to be significant drift. KSTest The KSTest metric calculates two Kolmogorov-Smirnov tests for each column to determine whether the data sets are derived from the same distributions. This metric measures the maximum distance between the empirical cumulative distribution functions (CDFs) of the data sets, without assuming any specific underlying distribution. KSTest is useful for detecting changes in distribution shape, location, and scale. This metric returns the probability that the distribution seen in the "real world" data is derived from the same distribution as the reference data. The closer the value is to 0, the more likely there is to be significant drift. ApproxKSTest The ApproxKSTest metric performs an approximate Kolmogorov-Smirnov test, ensuring that the maximum error is 6*epsilon compared to an exact KSTest. ApproxKSTest is useful for detecting changes in distributions for large data sets where performing an exact KSTest might be computationally expensive. This metric returns the probability that the distribution seen in the "real world" data is derived from the same distribution as the reference data. The closer the value is to 0, the more likely there is to be significant drift.

Commands:
oc login --token= <token> --server= <openshift_cluster_url>
TRUSTY_ROUTE=https://USD(oc get route/trustyai-service --template={{.spec.host}})
curl -H "Authorization: Bearer USDTOKEN" --location USDTRUSTY_ROUTE/q/openapi
curl -k -H "Authorization: Bearer USDTOKEN" -X POST --location USDTRUSTY_ROUTE/metrics/drift/meanshift/request --header 'Content-Type: application/json' --data <payload>
curl -k -H "Authorization: Bearer USDTOKEN" -X POST --location USDTRUSTY_ROUTE/metrics/drift/meanshift/request --header 'Content-Type: application/json' --data "{ \"modelId\": \"gaussian-credit-model\", \"referenceTag\": \"TRAINING\" }"
oc login --token= <token> --server= <openshift_cluster_url>
TRUSTY_ROUTE=https://USD(oc get route/trustyai-service --template={{.spec.host}})
curl -k -H "Authorization: Bearer USDTOKEN" -X GET --location "USDTRUSTY_ROUTE/metrics/drift/meanshift/requests"
curl -H "Authorization: Bearer USDTOKEN" -X GET --location "USDTRUSTY_ROUTE/metrics/all/requests"
curl -k -H "Authorization: Bearer USDTOKEN" -X DELETE --location "USDTRUSTY_ROUTE/metrics/drift/meanshift/request" -H "Content-Type: application/json" -d "{ \"requestId\": \"USDid\" }"
curl -H "Authorization: Bearer USDTOKEN" -X GET --location "USDTRUSTY_ROUTE/metrics/drift/meanshift/requests"