Configuring quality evaluations in watsonx.governance
Configuring quality evaluations in watsonx.governance watsonx.governance quality evaluations measure your foundation model's ability to provide correct outcomes When you [evaluate prompt templates](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-eval-prompt.html), you can review a summary of quality evaluation results for the text classification task type. The summary displays scores and violations for metrics that are calculated with default settings. To configure quality evaluations with your own settings, you can set a minimum sample size and set threshold values for each metric. The minimum sample size indicates the minimum number of model transaction records that you want to evaluate and the threshold values create alerts when your metric scores violate your thresholds. The metric scores must be higher than the threshold values to avoid violations. Higher metric values indicate better scores. Supported quality metrics When you enable quality evaluations in watsonx.governance, you can generate metrics that help you determine how well your foundation model predicts outcomes. watsonx.governance supports the following quality metrics: * Accuracy - Description: The proportion of correct predictions - Default thresholds: Lower limit = 80% - Problem types: Multiclass classification - Chart values: Last value in the timeframe - Metrics details available: Confusion matrix - Understanding accuracy: Accuracy can mean different things depending on the type of algorithm: - Multi-class classification: Accuracy measures the number of times any class was predicted correctly, normalized by the number of data points. For more details, see [Multi-class classification](https://spark.apache.org/docs/2.1.0/mllib-evaluation-metrics.htmlmulticlass-classification){: external} in the Apache Spark documentation. * Weighted true positive rate - Description: Weighted mean of class TPR with weights equal to class probability - Default thresholds: Lower limit = 80% - Problem type: Multiclass classification - Chart values: Last value in the timeframe - Metrics details available: Confusion matrix - Do the math: The True positive rate is calculated by the following formula:number of true positives TPR = _________________________________________________________ number of true positives + number of false negatives * Weighted false positive rate - Description: Weighted mean of class FPR with weights equal to class probability. For more details, see [Multi-class classification](https://spark.apache.org/docs/2.1.0/mllib-evaluation-metrics.htmlmulticlass-classification){: external} in the Apache Spark documentation. - Default thresholds: Lower limit = 80% - Problem type: Multiclass classification - Chart values: Last value in the timeframe - Metrics details available: Confusion matrix - Do the math: The Weighted False Positive Rate is the application of the FPR with weighted data.number of false positives FPR = ______________________________________________________ (number of false positives + number of true negatives) * Weighted recall - Description: Weighted mean of recall with weights equal to class probability - Default thresholds: Lower limit = 80% - Problem type: Multiclass classification - Chart values: Last value in the timeframe - Metrics details available: Confusion matrix - Do the math: Weighted recall (wR) is defined as the number of true positives (Tp) over the number of true positives plus the number of false negatives (Fn) used with weighted data.number of true positives Recall = ______________________________________________________ number of true positives + number of false negatives * Weighted precision - Description: Weighted mean of precision with weights equal to class probability - Default thresholds: Lower limit = 80% - Problem type: Multiclass classification - Chart values: Last value in the timeframe - Metrics details available: Confusion matrix - Do the math: Precision (P) is defined as the number of true positives (Tp) over the number of true positives plus the number of false positives (Fp).number of true positives Precision = ________________________________________________________ number of true positives + the number of false positives * Weighted F1-Measure - Description: Weighted mean of F1-measure with weights equal to class probability - Default thresholds: Lower limit = 80% - Problem type: Multiclass classification - Chart values: Last value in the timeframe - Metrics details available: Confusion matrix - Do the math: The Weighted F1-Measure is the result of using weighted data.precision * recall F1 = 2 * ____________________ precision + recall * Matthews correlation coefficient - Description: Measures the quality of binary and multiclass classifications by accounting for true and false positives and negatives. Balanced measure that can be used even if the classes are different sizes. A correlation coefficient value between -1 and +1. A coefficient of +1 represents a perfect prediction, 0 an average random prediction and -1 and inverse prediction. - Default thresholds: Lower limit = 80 - Chart values: Last value in the timeframe - Metrics details available: Confusion matrix * Label skew - Description: Measures the asymmetry of label distributions. If skewness is 0, the dataset is perfectly balanced, it if is less than -1 or greater than 1, the distribution is highly skewed, anything in between is moderately skewed. - Default thresholds: - Lower limit = -0.5 - Upper limit = 0.5 - Chart values: Last value in the timeframe Parent topic:[Configuring model evaluations](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-monitors-overview.html)