Horizon-Labs/multilingual-zeroshot-base

Multilingual Zero-Shot Classifier (base, 308M)

Classify text in 30+ languages into any labels you choose, with no training. Use it with the transformers zero-shot-classification pipeline, like facebook/bart-large-mnli, but multilingual, smaller, and with an 8k-token context window (fine-tuned at up to 1,024 tokens).

• Multilingual: the text can be in any of the languages below; labels and the hypothesis template stay in English.
• Commercially clean: Apache-2.0, trained only on data that allows commercial use (no XNLI, ANLI or other non-commercial sets). See Training.
• Small and fast: 308M parameters, ModernBERT architecture (mmBERT), ONNX included for CPU and the browser.
• Honest numbers: all models below were run by us with the same script and templates.

Try it in the browser: Horizon-Labs/multilingual-zeroshot demo.

Part of Horizon Labs' open models (collection). Source code: github.com/horizon-ai-labs/agent-io-guards.

Quick start

from transformers import pipeline

clf = pipeline("zero-shot-classification", model="Horizon-Labs/multilingual-zeroshot-base")
clf("Mi pedido llegó roto y quiero que me devuelvan el dinero.",
    candidate_labels=["refund request", "shipping question", "product praise", "account problem"])
# {'labels': ['refund request', ...], 'scores': [...]}

# several labels can apply at once
clf("The camera is great but the battery dies by noon.", ["camera", "battery", "screen", "price"], multi_label=True)

# a task-specific template often helps
clf("¿Me pones una alarma a las siete?", ["set an alarm", "play music", "weather"], hypothesis_template="The user wants to {}.")

Labels: not_entailment (0) and entailment (1). For each candidate label the model scores whether the text entails "This example is {label}." (or your hypothesis_template). Any NLI-style use works too: pass text and text_pair to a text-classification pipeline.

Evaluation

Accuracy, single-label (multi_label=False: the label with the highest entailment score wins). English templates and labels for every language; the same template for every model (e.g. "This text is about {}." for SIB-200). No model saw these datasets' training splits, except where marked. ‡ = trained partly on data with non-commercial licenses (their -c variants are the commercially usable ones). Script: zeroshot/evaluate_zs.py.

§ = label names seen in our synthetic training data. Since v1.1 our training data includes generic label taxonomies (topics, news sections, Q&A question topics, emotions, sentiment) whose label names overlap these benchmarks' label sets; for Yahoo Answers and AG News almost exactly. No benchmark texts were used, but on these rows our models are not zero-shot with respect to the label names, so compare with care. MASSIVE, Banking77 and XNLI label sets were not used.

Multilingual

	this model (308M)	small (141M)	large (568M)	bge-m3-zeroshot-v2.0-c (568M)	mDeBERTa-v3-base-xnli (278M)	xlm-roberta-large-xnli (560M)	bge-m3-zeroshot-v2.0 (568M) ‡
MASSIVE intents (60 labels), 16 languages	0.507	0.402	0.535	0.411	0.351	0.404	0.611
SIB-200 topics (7 labels), 16 languages §	0.819	0.789	0.834	0.782	0.654	0.526	0.837

Per language, mean of MASSIVE and SIB-200:

	this model (308M)	small (141M)	large (568M)	bge-m3-zeroshot-v2.0-c (568M)	mDeBERTa-v3-base-xnli (278M)	xlm-roberta-large-xnli (560M)	bge-m3-zeroshot-v2.0 (568M) ‡
English	0.708	0.654	0.716	0.596	0.537	0.504	0.752
German	0.666	0.595	0.685	0.607	0.527	0.472	0.748
French	0.707	0.650	0.704	0.617	0.526	0.487	0.753
Spanish	0.651	0.602	0.658	0.561	0.488	0.456	0.742
Portuguese	0.682	0.617	0.679	0.585	0.491	0.446	0.713
Russian	0.651	0.623	0.663	0.597	0.497	0.453	0.724
Polish	0.697	0.642	0.712	0.640	0.533	0.486	0.756
Turkish	0.685	0.601	0.680	0.605	0.491	0.455	0.713
Arabic	0.628	0.551	0.666	0.559	0.476	0.433	0.681
Hindi	0.602	0.541	0.675	0.589	0.511	0.459	0.719
Chinese	0.688	0.625	0.709	0.627	0.514	0.481	0.760
Japanese	0.714	0.651	0.695	0.637	0.530	0.492	0.752
Korean	0.670	0.578	0.707	0.610	0.494	0.486	0.714
Vietnamese	0.648	0.563	0.671	0.605	0.478	0.490	0.735
Indonesian	0.682	0.609	0.733	0.625	0.520	0.483	0.749
Swahili	0.531	0.425	0.595	0.484	0.430	0.359	0.573

English

	this model (308M)	small (141M)	large (568M)	bge-m3-zeroshot-v2.0-c (568M)	mDeBERTa-v3-base-xnli (278M)	xlm-roberta-large-xnli (560M)	bge-m3-zeroshot-v2.0 (568M) ‡	bart-large-mnli (407M)	deberta-v3-base-zeroshot-v2.0 (184M) ‡
AG News (4) §	0.824	0.828	0.814	0.726	0.670	0.591	0.886	0.684	0.884
Yahoo Answers (10) §	0.640	0.623	0.658	0.564	0.497	0.529	0.654	0.586	0.672
Banking77 (77)	0.555	0.534	0.623	0.430	0.287	0.166	0.695	0.480	0.714
Emotion (6) §	0.483	0.442	0.511	0.476	0.483	0.345	0.677	0.463	0.737
SST-2 (2) §	0.883	0.837	0.880	0.865	0.844	0.820	0.905	0.922	0.947
MASSIVE, English only	0.583	0.500	0.583	0.413	0.397	0.443	0.680	0.530	0.710
SIB-200, English only	0.833	0.809	0.848	0.779	0.676	0.564	0.824	0.760	0.745

Versions

v1.1 adds data with broad, reusable label taxonomies, so it is better on common categories (topics, emotions, sentiment, aspects) — the § rows, where the label names are familiar to it. On label sets it has not seen it stays within about ±0.015 of v1.0 (slightly lower on some).

v1.2 (this version) is distilled from the large model: it is trained on the same data, with half of the loss on the large model's probabilities instead of the hard labels. It gains most on the unseen MASSIVE label set; other rows move by about ±0.01. To pin an earlier model, load it with revision="v1.0" or revision="v1.1".

	v1.0	v1.1	v1.2 (this version)
MASSIVE (unseen label set)	0.492	0.482	0.507
Banking77 (unseen label set)	0.568	0.554	0.555
XNLI (balanced acc.)	0.801	0.799	0.792
SIB-200 §	0.798	0.813	0.819
AG News §	0.789	0.826	0.824
Yahoo Answers §	0.503	0.649	0.640
Emotion §	0.466	0.491	0.483
SST-2 §	0.884	0.875	0.883

NLI

	this model (308M)	small (141M)	large (568M)	bge-m3-zeroshot-v2.0-c (568M)	mDeBERTa-v3-base-xnli (278M)	xlm-roberta-large-xnli (560M)	bge-m3-zeroshot-v2.0 (568M) ‡
XNLI test, 12 languages (balanced acc.) †	0.792	0.760	0.800	0.825	0.845	0.992	0.818

† XNLI is included for reference only: xlm-roberta-large-xnli and mDeBERTa-xnli were trained on XNLI data (the first scores 0.99, which suggests it saw the test sentences). Our models never saw XNLI.

Limitations

• English-only models trained with more (partly non-commercial) classification data are better on English topic and emotion benchmarks (e.g. deberta-v3-base-zeroshot-v2.0 on Emotion and Yahoo). If you only need English, compare them on your data.
• bge-m3-zeroshot-v2.0 (568M, trained partly on non-commercial data) scores higher on MASSIVE and SIB-200.
• Zero-shot accuracy depends a lot on label wording and the template. Use descriptive labels ("request a refund" rather than "refund_req") and try a template that fits your task. The model links explicit wording better than implied categories. Example (small model, multi-label, a gym review not like our training domains): "The machines are always taken after 5pm and half the treadmills are broken, but the coaches really know their stuff. For 60 euros a month I expected cleaner showers." gives equipment 0.99, trainers 0.97, membership cost 0.82, but hygiene only 0.28 and crowding 0.03 (v1.0: trainers 0.56, membership cost 0.58).
• Broad labels (e.g. "world news", "education") tend to win over specific ones. Emotions close in meaning (joy / love / surprise) are often confused.
• With multi_label=True, scores are independent; tune the threshold on a few examples of your own.
• Lower-resource languages (e.g. Swahili) score clearly lower than high-resource ones.
• Much of the training data is synthetic (Qwen3.8-27B) or machine-translated.

Training

• Backbone: jhu-clsp/mmBERT-base (MIT), sequence-pair classification, bf16, max length 1024.
• Data (label = does the text entail the hypothesis):
  • English NLI: MultiNLI (OANC and CC-BY-SA-3.0 parts), SNLI (CC-BY-SA-4.0), WANLI (CC-BY-4.0).
  • 120k MultiNLI/WANLI pairs machine-translated by Qwen3.8-27B into 24 languages, with native and English hypotheses.
  • Synthetic zero-shot tasks by Qwen3.8-27B: FineWeb-Edu / FineWeb-2 passages (ODC-BY) labelled by topic, genre, audience, tone and purpose with near-miss wrong labels, and ~90k short texts (requests, reviews, tickets, posts, headlines) over 26 task types, 32 domains and 33 languages, each with an invented label set and hypothesis template.
  • (v1.2) Distillation: half of the loss uses the probabilities of multilingual-zeroshot-large (568M) on the same training pairs instead of the hard labels.
  • (v1.1) Generic taxonomies by Qwen3.8-27B: 24k new FineWeb / FineWeb-2 passages labelled for topic, text type, sentiment, audience, purpose and news section; ~130k short texts written for fixed label sets (emotion, sentiment, Q&A question topic, news section, customer-message topic, urgency, formality, spam) without using the label words; ~25k reviews in 8 domains mentioning aspects (e.g. "internet", "food") without naming them.
• The Qwen-generated classification data (short texts, taxonomies, aspects, labelled passages) is published as Horizon-Labs/multilingual-zeroshot-synthetic.
• Not used: XNLI, ANLI, FEVER-NLI, any benchmark above.