deepopen clinc_single
Author-reported frozen test accuracy: 97.7556% (4399/4500). This model card does not claim official leaderboard acceptance or SOTA certification.
CLINC150 closed-set classification over 150 in-scope intents: 15,000 training, 3,000 validation and 4,500 test examples. Out-of-scope examples and OOS rejection are excluded. Historical benchmark reuse occurred; these are retrospective frozen results, not a new independent test.
Dataset: CLINC oos-eval data_full.json, revision 828f8093932c8fe6ca7936c3d2e52903b1c523de, CC-BY-3.0. Prediction row indices follow its in-scope test order. Known normalized-text overlap is 2 train/test examples; the headline score uses the full official test. Training and validation contain 3 overlapping normalized texts.
Single Laya-derived encoder, R-Drop seed 42, classifier-only inference on masked mean-pooled states. The released selection was made on validation; prior-round test aggregates were already known. CUDA inference uses BF16 autocast and fixed-length padding. Frozen weight SHA256: a707c842bfd29311a3825d11152aab52423b159dcb4d2dfb740777db83577834.
Inference
Use Python 3.12 and a compatible NVIDIA CUDA environment. The existing release-loading environment is pinned in requirements-py312-lock.txt; these are loading versions, not historical training hardware. CUDA is required for the CLINC ensemble and full-test verification.
python -m pip install -r requirements.txt
python infer_clinc.py --checkpoint . --text "what is my bank balance"
Download the complete repository; custom Python files and configuration are required. Generic Transformers pipelines are not supported for custom CLINC releases. Prediction probabilities do not provide calibrated OOS rejection.
Reproduce and inspect
evaluation/predictions.csv provides every frozen prediction without redistributing utterances. evaluation/classes.csv provides per-class statistics. SHA256SUMS.json binds release files. Obtain the original dataset separately under its license, then run:
python verify_release.py --package clinc_single --checkpoint . --data /path/to/data_full.json --output reproduction.json
For BANKING77 the cached test JSONL schema is one object per official test row with text and label_text. The verifier checks row ordering against all gold labels and requires zero differences from the separately frozen current-runtime reference while recording the two historical differences. A failure must be reported, not used to tune the model. This retrospective reproducibility check does not create an independent benchmark result.
Base Laya: convaiinnovations/laya, revision 1c5edc17a7acd8701df6fc341c0d179f1c62c982, Apache-2.0. Preserve all bundled license and attribution files.
Evaluation explorer: https://huggingface.co/spaces/c41/laya-evaluation
Exact reproduction limitation (2026-10-21)
The unchanged released weight reproduces 4399/4500 (97.7556%) on the current A10 runtime, with TWO differences from the historical prediction vector: row 720 changes from accept_reservations to restaurant_reservation (now correct); row 3064 changes from ingredients_list to recipe (now incorrect). The cause has not been isolated. No model, precision, or backend was tuned against test outcomes.
The original predictions.csv and classes.csv remain historical evidence. reproduction_reference.csv records the full current-runtime prediction vector. reproduction_drift.json fixes its hash and these exact differences; verification requires an exact match to that reference and the disclosed historical differences. Equal aggregate accuracy alone does not pass. The original failed exact-reproduction audit is retained in historical_reproduction_drift.json. This release does not claim zero historical prediction drift.
- Downloads last month
- 15