deepopen banking_sam13
Author-reported frozen test accuracy: 94.1234% (2899/3080). This model card does not claim official leaderboard acceptance or SOTA certification.
A 77-class classifier continued with sharpness-aware minimization (SAM), seed 13, from the historical full-train checkpoint. Continuation uses all 10,003 official training rows, two fixed epochs (626 updates), batch 32, maximum length 128, encoder LR 2e-6, head LR 2e-5, rho 0.05, weight decay 0.01, and BF16 training. Evaluation uses eager FP32 SDPA, batch 32, maximum length 128.
The displayed checkpoint was selected AFTER observing three retrospective test results. It is the best observed seed, not an independent validation-selected improvement. SAM seeds 13/42/87 scored 2899/2896/2894 out of 3080 (mean 94.0368%). Paired confidence intervals include zero; no significant gain is established. The original historical 94.0909% result has unresolved exact reproduction drift (current eager FP32: 93.9935%). That historical weight is not this release's SAM13 weight. The former development subset was included in training and must not be treated as held out.
Frozen SAM13 weight SHA256: 0ff739f4413497119476af928b6cff482f1c0ef0cfe9bfb96527b769df172d29. Dataset: PolyAI/banking77, revision 18072d2685ea682290f7b8924d94c62acc19c0b2, CC-BY-4.0. Prediction row indices follow the archived official test ordering.
Inference
Use Python 3.12 and a compatible NVIDIA CUDA environment. The existing release-loading environment is pinned in requirements-py312-lock.txt; these are loading versions, not historical training hardware. CUDA is required for the CLINC ensemble and full-test verification.
python -m pip install -r requirements.txt
python infer_banking.py --checkpoint . --text "what is my bank balance"
Download the complete repository; custom Python files and configuration are required. Generic Transformers pipelines are not supported for custom CLINC releases. Prediction probabilities do not provide calibrated OOS rejection.
Reproduce and inspect
evaluation/predictions.csv provides every frozen prediction without redistributing utterances. evaluation/classes.csv provides per-class statistics. SHA256SUMS.json binds release files. Obtain the original dataset separately under its license, then run:
python verify_release.py --package banking_sam13 --checkpoint . --data /path/to/test.jsonl --output reproduction.json
For BANKING77 the cached test JSONL schema is one object per official test row with text and label_text. The verifier checks row ordering against all gold labels and requires zero differences from frozen predictions. A failure must be reported, not used to tune the model. This retrospective reproducibility check does not create an independent benchmark result.
Base Laya: convaiinnovations/laya, revision 1c5edc17a7acd8701df6fc341c0d179f1c62c982, Apache-2.0. Preserve all bundled license and attribution files.
Evaluation explorer: https://huggingface.co/spaces/c41/laya-evaluation
- Downloads last month
- 109