ITheEqualizer/persian-assistant-scenario-router

Persian Assistant Scenario Router v1.0.0

Persian Assistant Scenario Router is a compact, vocabulary-free classifier for advisory routing of Persian personal-assistant requests into 18 MASSIVE scenarios, with a validation-selected abstention threshold.

Maintainer: Ali Zakaee (ITheEqualizer).

Intended decision and audience

The named consumer is an offline Persian personal-assistant orchestrator that must repeatedly choose a capability before handling a request. The model maps one bounded fa-IR string to alarm, audio, calendar, cooking, datetime, email, general, iot, lists, music, news, play, qa, recommendation, social, takeaway, transport, or weather. A score below the fixed threshold returns abstention so the application can ask the user to clarify.

Representative positive decisions:

• فردا ساعت هفت صبح بیدارم کن should route to alarm.
• هوای تهران فردا چطور است؟ should route to weather.
• یک رستوران ایتالیایی پیشنهاد بده should route to recommendation.

A negative example is an empty string, which is rejected before inference. Ambiguous, mixed-language, or out-of-taxonomy requests should not be forced into a handler. The output is advisory: it does not execute tools, authorize actions, or replace application permission checks.

Tested installation and inference

python -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
.venv/bin/python -m persian_scenario_router.predict \
  "فردا ساعت هفت صبح بیدارم کن"
.venv/bin/python -m persian_scenario_router.reference_consumer \
  "هوای تهران فردا چطور است؟"

The reference consumer only reports the handler it would select. It never executes a handler. example.json records one correctly routed validation example and its expected score and decision.

Data, labels, rights, and leakage controls

Training uses the Persian fa-IR file from the authoritative Amazon MASSIVE v1.1 archive. The source release, extracted file, CC BY 4.0 license, and NOTICE are bound to SHA-256 values in dataset_manifest.json. MASSIVE localized the CC BY 4.0 SLURP assistant corpus. The source provides human-localized assistant utterances with scenario and intent labels; no model outputs were used as training labels.

The raw Persian file contains 16,521 rows. Ten conflicting normalized-template groups, covering 21 rows, were removed. The retained 16,500 rows include 500 duplicate-template groups; every group is contained in exactly one split. Stratified template-group folds assign 9,900 rows to training, 3,300 to validation, and 3,300 to the untouched lockbox, with all 18 classes in every split and zero template-group overlap.

Only 18 worker IDs exist, and one worker supplied about 42% of retained rows while omitting transport. A worker-isolated split could not preserve the full taxonomy. Results therefore control direct template leakage but may still benefit from annotator style shared across splits. This is a material limitation, not evidence of broad Persian-speaker generalization.

The audit found no URL-, email-, phone-, or secret-like strings under the recorded patterns. These are localized assistant requests, but downstream users must still treat real inputs as potentially sensitive and avoid logging them by default.

Representation, algorithm, and selection

Preprocessing applies Unicode NFKC normalization, case folding, whitespace collapse, an empty-input rejection, and a 512-character maximum. Features are fixed-width word 1–2 gram hashes (2,048 bins) and character-within-word 3–5 gram hashes (4,096 bins). An averaged SGDClassifier fits multinomial logistic scores with balanced class weights, L2 regularization (alpha=1e-5), seed 20260810, and a 500-epoch ceiling.

The strongest feasible compact baseline used learned word and character TF-IDF vocabularies and the same averaged SGD objective. It reached 0.8845 validation macro-F1 but serialized to 31.77 MB. The selected fixed-width model reaches 0.8813 validation macro-F1, a 0.0032 decrease, while reducing the artifact to 1.83 MB (94.2% smaller) and shipping no raw vocabulary. The measured deployment advantage cleared the predeclared 2 MB cap without an unacceptable quality or worst-slice regression.

The abstention threshold, 0.3432712760, was selected only on validation to maximize coverage subject to at least 0.90 routed accuracy. It achieved 98.12% validation coverage and 90.02% accuracy among routed examples. The untouched lockbox was opened only after every validation and artifact gate passed.

Measured results

Evaluation	Accuracy	Macro-F1	Worst recall	ROC-AUC	PR-AUC
Validation	0.8873	0.8813	0.5833 (general)	0.9910	0.9319
Template-group lockbox	0.8942	0.8896	0.6406 (general)	0.9914	0.9361
Learned-vocabulary validation baseline	0.8912	0.8845	0.5625 (general)	—	—

The template-group bootstrap 95% interval for lockbox macro-F1 is 0.8773–0.9003. Complete confusion counts and per-class precision, recall, F1, and support are in metrics.json.

The weakest lockbox scenario is general: 123/192 correct, 69 false negatives, and 74 false positives. cooking has 50/65 correct, while qa has 294/337 correct but attracts 72 false positives. These failures matter because a false route can invoke the wrong capability, while abstention adds clarification friction. Integrators should prefer clarification for high-impact or ambiguous requests and measure their own scenario costs.

Prediction trace

For فردا ساعت هفت صبح بیدارم کن, the loader verifies the model checksum and empty skops type set, NFKC-normalizes and case-folds the text, constructs its word and character hash vectors, and passes the 6,144 fixed features to the learned linear classifier. The largest scenario probability becomes the score. If it is at least 0.3432712760, the label is returned to the advisory reference router; otherwise the router returns ask_user_to_clarify.

Reproduction

Download the authoritative MASSIVE v1.1 archive, extract 1.1/data/fa-IR.jsonl, 1.1/LICENSE, and 1.1/NOTICE.md, then run:

python -m venv .train-venv
.train-venv/bin/python -m pip install -r requirements-train.txt
.train-venv/bin/python -m persian_scenario_router.release_train \
  --data /path/to/1.1/data/fa-IR.jsonl \
  --license /path/to/1.1/LICENSE \
  --notice /path/to/1.1/NOTICE.md \
  --output-dir reproduced \
  --reference reproduction_reference.npz

The clean reproduction must match every lockbox score, prediction, and label. The release build passed that check, checksum verification, secure serialization, static analysis, tests, and a fresh-environment inference quickstart.

Limitations and safe integration

• The 18 broad scenarios do not cover every assistant request and are not a permission or tool-selection policy by themselves.
• Persian dialects, code switching, misspellings, speech-recognition errors, new intents, and long conversational context are underrepresented.
• Template grouping prevents duplicate leakage, but worker style is shared across splits because a worker-isolated full-taxonomy split is infeasible.
• Model probabilities are ranking scores, not guarantees of correctness or calibrated real-world risk.
• Hash collisions trade interpretability for a small, vocabulary-free artifact.
• Do not automatically execute sensitive actions. Apply independent permission, schema, provenance, and confirmation checks after routing.

Rollback is simple: disable the classifier and send every request to the existing clarification or manual-routing path. Before loading, verify model.skops against config.json; use skops with trusted=[]; reject checksum, schema, label-map, empty-input, and length mismatches.