Report for ProsusAI/finbert
Hi Team,
This is a report from Giskard Bot Scan 🐢.
We have identified 1 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_allagree, split train).
👉Robustness issues (1)
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 6.2% of the cases. We expected the predictions not to be affected by this transformation.
| Level | Metric | Transformation | Deviation |
|---|---|---|---|
| medium 🟡 | Fail rate = 0.062 | Add typos | 62/1000 tested samples (6.2%) changed prediction after perturbation |
Taxonomy
avid-effect:performance:P0201🔍✨Examples
| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 1919 | Cash flow from operations totalled EUR 7.4 mn , compared to a negative EUR 68.6 mn in the second quarter of 2008 . | Cash dlow from operations totalled WUR 7.4 mhn , comlared to a negative EUR 68.6 mn in the second wquarter of 2008 . | positive (p = 0.86) | negative (p = 0.95) |
| 1584 | `` These developments partly reflect the government 's higher activity in the field of dividend policy . '' | `` These developmsents partly reflect he government 's higher activity in the field of dividend policty . '' | positive (p = 0.75) | neutral (p = 0.58) |
| 628 | The share of the share capital of both above mentioned shareholders remains below 5 % . | The share of the share capiral of both above mentioned sharehooders remains below 5 % . | neutral (p = 0.72) | negative (p = 0.83) |
Checkout out the Giskard Space and Giskard Documentation to learn more about how to test your model.
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.