Horizon-Labs/multilingual-question-vs-statement-small

Multilingual Question vs Statement Classifier (small, 141M)

Tells whether a text is a question or a statement / keyword query, in English and 65 other languages, without needing a question mark (ASR transcripts, typed search queries). A multilingual drop-in replacement for shahrukhx01/question-vs-statement-classifier, Haystack's default query router: same labels, LABEL_1 = question, LABEL_0 = statement or keywords. Built on mmBERT-small, Apache-2.0, ONNX for CPU and the browser (transformers.js) included. onnx/model_quantized.onnx (int8 embeddings, 268 MB) gives the same label as fp32 for 100.0% of 400 test texts.

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="Horizon-Labs/multilingual-question-vs-statement-small")
print(clf(["how do I reset my password", "best pizza in new york", "¿dónde está la estación", "ich habe den zug verpasst"]))
# LABEL_1 = question, LABEL_0 = statement / keywords

In Haystack, use it wherever shahrukhx01/question-vs-statement-classifier is configured (for example as the model of a TransformersTextRouter); the label names are the same, so existing routes keep working.

Evaluation

Labels in both test sets are objective, not judged by a model. No test text was used for training.

• Web sentences: 11,357 held-out sentences from FineWeb-2 / FineWeb (66 languages, up to 100 questions and 100 statements each). A sentence is a question if it ended in a question mark (?, ？, ؟, Greek ;, Armenian ՞). The mark and other final punctuation were removed and half the sentences lowercased. Balanced accuracy is averaged over the 62 languages with at least 20 questions (Thai, Burmese and Khmer rarely use question marks).
• Search queries: 9,399 held-out search queries in 47 languages, natural-language questions (question mark removed) vs keyword queries, generated by Qwen3.8-27B from web passages.

model	web sentences: mean balanced accuracy over languages	web sentences: macro-F1	web sentences, English: accuracy	search queries, 47 languages: accuracy	search queries, English: accuracy	note
this model (141M)	0.928	0.932	0.972	0.990	0.995	66 languages
shahrukhx01/question-vs-statement-classifier (11M)	0.515	0.405	0.893	0.519	1.000	English (bert-mini); Haystack's default

• The table shows the released checkpoint. Means over 2 training seeds (sentences, mean balanced accuracy .928 / .928; queries .990 / .989).
• Lowest languages (balanced accuracy): ita_Latn 0.84, hye_Armn 0.85, dan_Latn 0.86, slk_Latn 0.87, por_Latn 0.87, hrv_Latn 0.88. Rhetorical questions and questions with no interrogative word or word order (intonation questions such as "you're coming tomorrow") cannot be recognised from text alone.

Training

• 299,322 real sentences from FineWeb-2 and FineWeb (ODC-BY) in 67 languages, labelled by their question mark, which was then removed; 112,753 generated search queries (question vs keywords). No test sentences or queries.
• mmBERT-small, two classes, max length 128 tokens. Code: code/ in this repository.

Limitations

• Labels follow punctuation: rhetorical questions count as questions and questions written without a question mark in the source count as statements, so some "errors" are label noise.
• Plain imperatives ("open the window") are statements here, as in the original Haystack model, but polite requests for information ("can you tell me ...", Japanese "...教えてください") often come out as questions. Check this on your own traffic if the difference matters.