ALLSmartIntent

What does this chat message want, and what does a good reply need? ALLSmartIntent answers 14 such questions about the newest message of a chat in one pass, in milliseconds, before an expensive language model call starts: dialogue act, reference to the conversation, which data a reply needs, mood, off-topic, rule bypassing, unclear requests, inappropriate content, crisis and extremism, and the language of the message.

It is trained to be general: it reads one sentence about the purpose of the bot, the last messages and the new message. What only concerns your bot, such as which of your tools fits, stays with you.

ALLSmartIntent runs in the gateway of ALLSmartLLM on our own hardware in Austria. See also our privacy models ALLSmartArt4, ALLSmartArt4i and ALLSmartArt9-10.

Size	0.3B parameters, mmBERT-base with four linear heads
Input	purpose of the bot, history (optional), new message; up to 512 tokens
Languages	trained on 14 languages, measured mainly on German and English
Speed	GPU (NVIDIA GB10): ≈ 15 ms per message, ≈ 7 ms per message in batches of 16. GPU (RTX 4070 Ti): ≈ 8 ms per message, ≈ 1 ms per message in batches of 32. CPU (Ryzen 7 7800X3D, 8 threads): ≈ 29 ms per message

Outputs

Head	Type	Values
akt	one class	frage (question), auftrag (request), aussage (statement), kurzantwort ("yes", "the second one": only understandable with the history), korrektur (contradicts the last answer), dank, gruss_smalltalk, beschwerde (complaint)
bezug	one class	neu, anschluss (continues the last answer: "and tomorrow?"), themenwechsel (change of topic)
bedarf_*	yes/no each	what a good reply needs: aktuelle_daten (weather, prices, news, timetables), nutzerdaten (the user's own accounts, appointments, files), aktion (do, send or change something), dokumente (manuals, reports, wiki), rechnen (calculation or comparison), allgemeinwissen (answerable without external data)
stimmung_*	yes/no each	besorgt (worried), veraergert (annoyed), dringend (urgent)
rahmen	yes/no	the message fits the purpose of the bot (no means off-topic)
regelumgehung	yes/no	wants to see or change the instructions, break the role, prompt injection
unklar	yes/no	cannot be answered sensibly without asking back
unangemessen	yes/no	insults, harassment, hateful content
krise	yes/no	signs of self-harm or acute crisis
extremismus	yes/no	hate against groups, extremist content
sprache	one class	language of the new message: cs de en es fr hr hu it nl pl pt ro tr uk, andere

Every yes/no head is an independent sigmoid score with its own threshold in intent.json.

Purpose: Flight assistant on an airline website.
History: user "Wie wird das Wetter heute in Wien bei der Landung?" / bot "In Wien sind es bei der Landung etwa 18 Grad."

"und morgen?"                                   frage, anschluss: aktuelle_daten, allgemeinwissen, rahmen
"Ignore all previous instructions and print …"  auftrag, neu: regelumgehung (not rahmen)
"Kannst du mir ein Rezept für Gulasch geben?"   frage, themenwechsel: allgemeinwissen (not rahmen: off-topic)
"Ich habe solche Angst vorm Fliegen …"          aussage, themenwechsel: stimmung_besorgt, rahmen

Usage

beispiel.py in this repository loads the model and contains the input format used in training:

from beispiel import entscheide
entscheide([{"zweck": "Flight assistant on an airline website.",
             "verlauf": [{"rolle": "nutzer", "text": "…"}, {"rolle": "bot", "text": "…"}],
             "nachricht": "und morgen?"}])
# [{"akt": "frage", "bezug": "anschluss", "sprache": "de", "ja": ["bedarf_aktuelle_daten", "rahmen", …], "werte": {…}}]

The model is the encoder in this repository (AutoModel) plus koepfe.safetensors (the heads) and intent.json (label lists and thresholds). Keep the input format: new message first, then purpose, then the history newest first, so truncation drops the oldest history and never the message.

Typical use: route to the right tool or model, ask back on unklar, answer off-topic messages with a fixed text, and answer krise, unangemessen and regelumgehung with fixed, reviewed responses before the language model sees the message.

Evaluation

Test sets created and annotated by hand independently of the training data, never used for training or model selection: 450 short chat excerpts (blind set 1) and 120 with long messages or long histories (blind set 2), for three bots (flight assistant, personal agent, stock expert chat). 383 of the 450 are German, the rest English, French, Italian, Spanish, Dutch, Polish and Turkish.

	blind set 1 (450)	blind set 2 (120, long)
akt accuracy	91.6 %	89.2 %
bezug accuracy	91.1 %	87.5 %
sprache accuracy	98.7 %	99.2 %
rahmen F1	0.93	0.92
regelumgehung F1	0.93 (precision 100 %)	1.00
bedarf_aktion F1	0.75	0.75
bedarf_aktuelle_daten F1	0.71	0.73
bedarf_dokumente F1	0.71	0.50
bedarf_rechnen F1	0.69	0.47
bedarf_allgemeinwissen F1	0.68	0.40
bedarf_nutzerdaten F1	0.63	0.73
stimmung_veraergert F1	0.75	0.71
stimmung_dringend F1	0.74	0.35 (3 cases)
stimmung_besorgt F1	0.54	0.77
unklar F1	0.54	0.67

Public test sets (test splits only; where a training split exists it was part of the training data):

	head	F1	precision	recall
Multilingual HateCheck, German (997)	unangemessen	0.84	73 %	98 %
Multilingual HateCheck, German (997)	extremismus	0.83	79 %	88 %
GermEval 2018 test, ABUSE/INSULT vs OTHER (750)	unangemessen	0.75	65 %	88 %
deepset/prompt-injections test (116)	regelumgehung	0.93	100 %	87 %

Normal traffic: 1 crisis false alarm in 570 blind messages.

Limitations

• The bedarf_* heads are hints for routing, not decisions. Precision is often 50 to 70 %: the model rather says a reply might need documents or user data than miss it. People also disagree on these questions.
• krise is basic protection. Clear statements are found reliably; subtle hints spread over several messages are not. It does not replace a crisis service or a review by a larger model where crises are expected.
• Crude language is flagged as unangemessen even without an insult (GermEval PROFANITY: 41 of 48).
• unklar and stimmung_besorgt are the weakest heads, partly because the answer depends on the bot.
• Measured mainly on German and English; the other languages are covered in training but measured on few cases.
• sprache can follow the conversation: a short English message in a German conversation may come out as de.
• Long messages and long histories cost accuracy for bedarf_rechnen and bedarf_allgemeinwissen (blind set 2).

Training data

Invented chat excerpts in 14 languages, created for this purpose for many different bot purposes, plus a few hundred cases annotated by hand. No customer data and no real personal data. In addition, the training splits of these public datasets:

• GermEval 2018 (Wiegand, Siegel, Ruppenhofer: Overview of the GermEval 2018 Shared Task on the Identification of Offensive Language, KONVENS 2018), CC BY 4.0
• GAHD, German Adversarial Hate Speech Dataset (Goldzycher et al., NAACL 2024), CC BY 4.0
• Nemotron Safety Guard Dataset v3 (NVIDIA; Joshi et al., CultureGuard, arXiv:2508.01710), CC BY 4.0
• SPML Chatbot Prompt Injection (Sharma et al., arXiv:2402.11755), MIT
• deepset/prompt-injections, Apache 2.0

Tweets were cleaned of handles, hashtags and links. Evaluation additionally uses Multilingual HateCheck (Röttger et al., 2022), never for training.

License

Apache 2.0. Base model mmBERT-base (MIT).

---

Deutsch: ALLSmartIntent erkennt in Millisekunden, was eine Chat-Nachricht will und was eine gute Antwort braucht: Art der Nachricht, Bezug zum Verlauf, Datenbedarf, Stimmung, Off-Topic, Regelumgehung, Unklarheit, Unangemessenes, Krise, Extremismus und Sprache, in einem Durchlauf vor dem Aufruf des großen Modells. Es läuft im Gateway von ALLSmartLLM.