tinydistill-massive-da-multilingual

A small text classifier distilled from Qwen/Qwen3.5-9B,google/gemma-4-12B-it into sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 with tinydistill, exported to ONNX for fast CPU inference.

Labels (60): alarm_query, alarm_remove, alarm_set, audio_volume_down, audio_volume_mute, audio_volume_other, audio_volume_up, calendar_query, calendar_remove, calendar_set, cooking_query, cooking_recipe, datetime_convert, datetime_query, email_addcontact, email_query, email_querycontact, email_sendemail, general_greet, general_joke, general_quirky, iot_cleaning, iot_coffee, iot_hue_lightchange, iot_hue_lightdim, iot_hue_lightoff, iot_hue_lighton, iot_hue_lightup, iot_wemo_off, iot_wemo_on, lists_createoradd, lists_query, lists_remove, music_dislikeness, music_likeness, music_query, music_settings, news_query, play_audiobook, play_game, play_music, play_podcasts, play_radio, qa_currency, qa_definition, qa_factoid, qa_maths, qa_stock, recommendation_events, recommendation_locations, recommendation_movies, social_post, social_query, takeaway_order, takeaway_query, transport_query, transport_taxi, transport_ticket, transport_traffic, weather_query

Usage

from tinydistill import Predictor  # pip install tinydistill

clf = Predictor("hf:Shroomy/tinydistill-massive-da-multilingual")
clf.predict(["some text"])

Or from the command line:

tinydistill predict hf:Shroomy/tinydistill-massive-da-multilingual "some text"
tinydistill serve hf:Shroomy/tinydistill-massive-da-multilingual

Results

Model	Device	Accuracy	Macro-F1	Top-3 accuracy	Agreement with teacher	Latency, batch 1 (ms)	Latency, batch 32 (ms)	Throughput (texts/s)	Size (MB)	Speed-up, batch 1
teacher ensemble of 2, 16 similar examples, 4-bit	cuda:0	0.806	0.779	0.910	n/a	n/a	n/a	n/a	23920	n/a
teacher, 16 similar examples, 4-bit: Qwen3.5-9B	cuda:0	0.781	0.756	0.896	n/a	n/a	n/a	n/a	n/a	n/a
teacher, 16 similar examples, 4-bit: gemma-4-12B-it	cuda:0	0.798	0.771	0.887	n/a	n/a	n/a	n/a	23920	n/a
student, PyTorch	cpu	0.811 ± 0.002	0.783 ± 0.002	0.922 ± 0.002	0.867 ± 0.003	8.3	136.8	233.9	95	n/a
student, ONNX fp32	cpu	0.811	0.781	0.921	0.863	5.8	125.0	256.0	96	n/a
student, ONNX int8	cpu	0.810	0.787	0.921	0.857	1.9	46.2	692.8	24	n/a

Task massive-da, 2974 gold-labelled test texts. Teacher: Qwen/Qwen3.5-9B,google/gemma-4-12B-it. Student: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2. The teacher's prompt shows each text the 16 most similar of 476 gold training examples. The student learns from the teacher's soft labels on all 11514 training texts, plus cross-entropy on 476 gold-labelled texts (8 per class; weight 1.0). Values with ± are the mean and standard deviation over 3 training seeds; the ONNX rows are the exported first-seed model. CPU rows use 2 threads. Latency is the median over repeated runs. Size is the teacher's weights as loaded and the students' model files. The teacher packs several texts into one forward pass; it is timed at batch 32 only when that takes under three minutes, otherwise its throughput is 1 / batch-1 latency.

Cascade: student first, unsure texts to the teacher

Share sent to the teacher	Student confidence threshold	Accuracy	Macro-F1	Mean latency per text (ms)
0%	0.129	0.810	0.787	n/a
10%	0.546	0.814	0.789	n/a
20%	0.734	0.817	0.794	n/a
30%	0.864	0.816	0.794	n/a
50%	0.951	0.807	0.781	n/a
100%	1.000	0.806	0.779	n/a

The student answers every text and passes its least confident ones to the teacher. Thresholds are set on the test texts to hit each share; in deployment, set them on a validation split. Latency is the student's batch-1 latency plus the share times the batch-1 latency of the teacher.

License

Check the licences of the student model, the teacher and the training data before using this model. The uploader chose cc-by-4.0.