mmbert-small-support-triage-extra-light

mmbert-small-support-triage cut down for phones and low-memory devices: the same classifier, labels and recipe, with 12 of mmBERT-small's 22 layers instead of 16 and a vocabulary pruned to the ten languages it was trained on. A 47 MB download and about 190 MB of browser memory, against 55 MB and about 240 MB for the 16-layer model with the same tokenizer. Routes a customer message to one of 27 request types in 11 departments of an online shop, or to out_of_scope. Ten languages: en, pl, de, fr, es, it, nl, pt, cs, sv. Normalise the text before calling it, as below.

Code, data pipeline and benchmark: github.com/balticml/support-triage.

Use

import { pipeline } from "@huggingface/transformers";
const classify = await pipeline("text-classification", "balticml/mmbert-small-support-triage-extra-light", { dtype: "q8", device: "wasm" });
const normalise = (t) => t.toLowerCase().replace(/[^\P{P}']/gu, "").replace(/\s+/g, " ").trim();
await classify(normalise("Hi, where is my order #48213? It was supposed to arrive on Monday."), { top_k: 3 });

Results

	test split (5,000)	160 hand-written messages	not-a-request recall	download	browser memory
this model: 12 layers, int8 44 MB, pruned tokenizer 3 MB	99.1	91.9	98.3	47 MB	187 MB
the same model with the full tokenizer	99.1	93.8	98.2	78 MB	—
the 16-layer model, pruned tokenizer (v1.3.0)	99.3	96.9	98.6	55 MB	238 MB
the 16-layer model, full tokenizer (v1.2.0)	99.3	96.9	98.4	86 MB	307 MB

Accuracy in %, all rows measured the same way: the int8 ONNX files in ONNX Runtime, 500 test requests and 500 not-a-request sentences per locale (seed 0). The 16-layer model's own card reports 96.2 on the hand-written messages in PyTorch fp32; int8 gets one message more right. Memory: Chrome, performance.measureUserAgentSpecificMemory() after loading the model, sorting eight messages and classifying thirty more, WASM with one thread; it does not grow with use. The hand-written messages were written by the author of the training messages, so they are an upper bound for a real inbox.

The pruned tokenizer

The full mmBERT tokenizer has 256,000 tokens and 580,604 merges and takes about 90 MB in a browser. The model only knows the 46,681 tokens seen in training; every other id becomes [UNK] inside the graph. The pruned tokenizer keeps those tokens, every token BPE forms on the way to one of them, the byte-fallback and special tokens, and the merges between them: 51,032 tokens and 128,416 merges, ids unchanged. For any text whose full tokenization uses only known tokens the result is identical, which held for all 9,819 such texts checked (test split, not-a-request sentences, hand-written messages, 5,000 made-up strings). A word the model never saw is split into smaller known pieces instead of becoming [UNK]; on the hand-written messages, where 116 of 160 contain such a word, that costs 3 of 160.

Limits

As the 16-layer model: nine of the ten test languages are machine translations, one e-commerce taxonomy, one label per message. Three points lower on the hand-written messages.

Licences

Model: Apache-2.0. mmBERT: Apache-2.0. Bitext: CDLA-Sharing-1.0. MADLAD-400: Apache-2.0. FLORES-200: CC BY-SA 4.0. Tatoeba: CC BY 2.0.