Model vulnerabilities detected
#14
by
giskard-bot
- opened
Hi, this is a report generated by the Giskard Bot.
We have identified 5 potential vulnerabilities in your model based on an automated scan. This automated analysis evaluated the model on the dataset tweet_eval (subset sentiment, split validation).
However, it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
Report summary
Robustness issues (5)
| Level | Metric | Transformation | Deviation | Description |
|---|---|---|---|---|
| Major | Fail rate = 0.147 | Transform to uppercase | 14.7% of tested samples changed prediction after perturbation | When we perturb the content of feature “text” with the transformation “Transform to uppercase”, your model changes its prediction in about 14.7% of the cases. We expected the predictions not to be affected by this transformation. |
| Major | Fail rate = 0.128 | Add typos | 12.8% of tested samples changed prediction after perturbation | When we perturb the content of feature “text” with the transformation “Add typos”, your model changes its prediction in about 12.8% of the cases. We expected the predictions not to be affected by this transformation. |
| Medium | Fail rate = 0.092 | Transform to title case | 9.2% of tested samples changed prediction after perturbation | When we perturb the content of feature “text” with the transformation “Transform to title case”, your model changes its prediction in about 9.2% of the cases. We expected the predictions not to be affected by this transformation. |
| Medium | Fail rate = 0.082 | Punctuation Removal | 8.2% of tested samples changed prediction after perturbation | When we perturb the content of feature “text” with the transformation “Punctuation Removal”, your model changes its prediction in about 8.2% of the cases. We expected the predictions not to be affected by this transformation. |
| Medium | Fail rate = 0.052 | Transform to lowercase | 5.2% of tested samples changed prediction after perturbation | When we perturb the content of feature “text” with the transformation “Transform to lowercase”, your model changes its prediction in about 5.2% of the cases. We expected the predictions not to be affected by this transformation. |
View full report
Debug your issues in the Giskard Hub
This is an automated Discussion created by the Giskard Bot.