mmbert-small-massive-intent

mmBERT-small fine-tuned on MASSIVE 1.1 (60 intents, 52 locales) plus a 61st class, out_of_scope, for text that is not an assistant command; exported to ONNX for transformers.js. Inputs are lower-cased and stripped of punctuation before tokenisation (the widget does this). Code, configs and benchmark: balticml/intent-detection. Demo: balticml.eu/demos/intent-detection.

import { pipeline } from "https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.3.1/dist/transformers.min.js";
const classify = await pipeline("text-classification", "balticml/mmbert-small-massive-intent", { dtype: "q8" });
await classify("Skasuj alarm na jutro rano", { top_k: 3 });

Results

Accuracy (%) on the MASSIVE test split. LLM rows: label list in the prompt, 200 test utterances per locale.

	en-US	pl-PL	de-DE	fr-FR	ja-JP	mean of 52 locales
this model, 84M params	88.4	86.2	86.3	87.3	87.2	85.1
Gemini 3.5 Flash-Lite, zero-shot	84.5	85.0	83.5	81.5	82.0	81.5
Qwen2.5-7B-Instruct Q4_K_M, zero-shot	63.0	60.5	62.0	62.0	63.0	50.5

Not-a-command recall on held-out FLORES-200 and Tatoeba sentences: 94.8% (mean of 52 locales); 1.5% of MASSIVE test commands are rejected. The int8 file agrees with fp32 on 399 of 400 parity utterances. One utterance takes 4.2 ms in ONNX Runtime on four CPU threads.

Files

file		size
onnx/model_quantized.onnx	weight-only int8, vocabulary pruned to 108,659 rows (ids remapped in the graph)	88 MB
tokenizer.json, configs	unchanged mmBERT tokenizer	34 MB

Training: all 52 MASSIVE train splits plus 1,500 non-command sentences per locale (FLORES-200 dev, Tatoeba; loss weight 0.3), one typo or one swapped word on 30% of utterances, 4 epochs, lr 5e-5, batch 64, dropout 0.1, label smoothing 0.1, weight decay 0.05, bf16, seed 42, best validation epoch. Built for short assistant-style commands in the 52 MASSIVE locales; other domains and languages will score lower. MASSIVE has no timer intent. Tokens outside the pruned vocabulary map to UNK.

Licences: Apache-2.0; mmBERT Apache-2.0; MASSIVE CC BY 4.0.