deepopen clinc_ensemble
Author-reported frozen test accuracy: 98.0222% (4411/4500). This model card does not claim official leaderboard acceptance or SOTA certification.
CLINC150 closed-set classification over 150 in-scope intents: 15,000 training, 3,000 validation and 4,500 test examples. Out-of-scope examples and OOS rejection are excluded. Historical benchmark reuse occurred; these are retrospective frozen results, not a new independent test.
Dataset: CLINC oos-eval data_full.json, revision 828f8093932c8fe6ca7936c3d2e52903b1c523de, CC-BY-3.0. Prediction row indices follow its in-scope test order. Known normalized-text overlap is 2 train/test examples; the headline score uses the full official test. Training and validation contain 3 overlapping normalized texts.
Four encoders: 0.5 x Laya probabilities + (DeBERTa seed13 + seed42 + seed87 probabilities)/6. This is an ensemble, not a single-model result. CUDA inference uses BF16 autocast and fixed-length padding. See system.json for frozen members. DeBERTa components use MIT; see LICENSE_DEBERTA.
Inference
Use Python 3.12 and a compatible NVIDIA CUDA environment. The existing release-loading environment is pinned in requirements-py312-lock.txt; these are loading versions, not historical training hardware. CUDA is required for the CLINC ensemble and full-test verification.
python -m pip install -r requirements.txt
python infer_hybrid.py --checkpoint . --text "what is my bank balance"
Download the complete repository; custom Python files and configuration are required. Generic Transformers pipelines are not supported for custom CLINC releases. Prediction probabilities do not provide calibrated OOS rejection.
Reproduce and inspect
evaluation/predictions.csv provides every frozen prediction without redistributing utterances. evaluation/classes.csv provides per-class statistics. SHA256SUMS.json binds release files. Obtain the original dataset separately under its license, then run:
python verify_release.py --package clinc_ensemble --checkpoint . --data /path/to/data_full.json --output reproduction.json
For BANKING77 the cached test JSONL schema is one object per official test row with text and label_text. The verifier checks row ordering against all gold labels and requires zero differences from frozen predictions. A failure must be reported, not used to tune the model. This retrospective reproducibility check does not create an independent benchmark result.
Base Laya: convaiinnovations/laya, revision 1c5edc17a7acd8701df6fc341c0d179f1c62c982, Apache-2.0. Preserve all bundled license and attribution files.
Evaluation explorer: https://huggingface.co/spaces/c41/laya-evaluation