PermitGuard — PTW Risk Classifier (TF-IDF + Logistic Regression + Keyword Prior)
Text-only bilingual (FA/EN) risk-level and work-type classifiers for synthetic Permit-to-Work
narratives. Ground-truth risk comes from 13 deterministic, auditable rules on structured fields;
the model sees only free text, so it is scored on whether it can recover that rule-derived label
(including the designed ~19% text/flag mismatches). Part of Aria AI
Aria SafeOps → Control of Work.
Data honesty
Trained only on permitguard-ptw-samples
— 100% synthetic, no real permits. Do not treat the numbers below as field PTW performance.
Held-out results (125 permits)
| Task | Metric | Value |
|---|---|---|
| Risk level (low/medium/high/critical) | Accuracy | 88.8% |
| Risk level | Macro-F1 | 0.866 |
| Risk level | Critical recall | 92.0% (23/25) |
| Work type (7 classes) | Accuracy | 100% |
Work-type accuracy is 100% because the synthetic narratives name the work type in the first sentence (by design, for a readable demo). It is not a claim that work-type NER is solved in the field. Critical recall is the headline safety metric; the two misses are consistent with the designed text/flag mismatch slice.
SIMOPS is a deterministic rule engine, not this sklearn model. 260 pairs, 50% conflict, five rules (SIMOPS-01…05) each fired 26 times. Precision/recall vs those labels would be circular 100% and is therefore not reported as a learned KPI.
Files
tfidf_vectorizer.joblib— word (1–2gram) + char_wb (3–5gram) TF-IDFrisk_classifier.joblib— multinomial Logistic Regression (balanced)worktype_classifier.joblib— same features → 7 work typeseval_results.json/benchmark_report.json/manifest.json
Inference also applies a small keyword prior (ContractGuard-style): if the text explicitly says a
critical precaution was not done, critical probability is boosted. See src/permitguard/models.py.
Related
- Dataset: alirezaaminzadeh/permitguard-ptw-samples
- Space: alirezaaminzadeh/permitguard-ptw-risk-classifier
- Sibling RAG: alirezaaminzadeh/petrosafe-rag-fa
- Product: aria-ai.ir
MIT · Aria AI Engineering Team
- Downloads last month
- -
Space using alirezaaminzadeh/permitguard-risk-classifier 1
Evaluation results
- accuracy on PermitGuard synthetic PTW held-out test (125 permits)self-reported0.888
- macro_f1 on PermitGuard synthetic PTW held-out test (125 permits)self-reported0.866
- critical_recall on PermitGuard synthetic PTW held-out test (125 permits)self-reported0.920