Swift-Support/labse-priority-1.0

Swift-Support LaBSE Priority Classifier (v1.0)

A fine-tuned LaBSE (Language-Agnostic BERT Sentence Embedding) model that assigns an escalation priority — Low / Medium / High — to a banking support ticket written in any of five language tracks. Built for the Swift support-ticket triage project, alongside Swift-Support/labse-intent-1.0.

Model details

• Base architecture: sentence-transformers/LaBSE (471M parameters, 501k vocabulary)
• Task: 3-class text classification (priority / urgency)
• Classes: Low, Medium, High
• Languages: English, Sinhala, Tamil, Singlish (romanized Sinhala), Tanglish (romanized Tamil)
• Regime: one multilingual model over all five tracks — not five per-language models

The headline metric is macro-F1, never accuracy: the class distribution is roughly 55% Low / 36% Medium / 9% High, so accuracy flatters a model that neglects High.

Evaluation

Trained on train+dev (49,990 rows = 9,998 tickets × 5 languages), scored once on the held-out test set (15,395 rows = 3,079 tickets × 5 languages). Frozen split e7b5934392cd; test tickets come from the official BANKING77 test file and were never used for model selection.

Pooled test macro-F1: 0.8901 (accuracy 0.9008)

Per-class F1: Low 0.9206 · Medium 0.8760 · High 0.8735

Against the alternatives (pooled test macro-F1)

model	macro-F1
gemma-3-1b multitask (shared head)	0.8904
LaBSE (this model)	0.8901
gemma-3-1b (LoRA, single-task)	0.8898
mmBERT	0.8887
XLM-RoBERTa base	0.8872
TF-IDF + LinearSVC (classical champion)	0.8722
TF-IDF + logistic regression	0.8683

The classical champion's 95% CI is [0.8605, 0.8831], so this model clears its upper bound — a real, if modest, win. The Gemma multitask variant is a statistical tie, not a better model.

Per language, on test

track	LaBSE (this model)	classical TF-IDF	delta
English	0.9229	0.9032	+0.0197
Sinhala	0.9179	0.8745	+0.0434
Singlish (romanized)	0.8817	0.8915	−0.0098
Tamil	0.9130	0.8905	+0.0225
Tanglish (romanized)	0.8142	0.7994	+0.0148
ALL (pooled)	0.8901	0.8722	+0.0179

Two things worth stating plainly:

1. The classical baseline still wins on Singlish. LaBSE gives back most of its native-script advantage on romanized text — a pattern that also shows up in linear probing, where LaBSE has the largest native-minus-romanized gap of any backbone in the roster.
2. Tanglish is the weak track, 7–10 points below every other language for both model families.

⚠️ The label ceiling — read this before quoting 0.89

The training labels were generated by an LLM prompt, not by human annotators. On a 500-ticket benchmark set that was annotated by hand, those prompt labels agree with human judgement at 0.7722 macro-F1 (95% CI [0.7263, 0.8147]; raw agreement 0.804, Cohen's κ = 0.644).

This does not cap the number above — against the prompt labels a model could in principle reach 1.0. It caps what the number means. This model has learned the labeling rule well; the rule itself agrees with a human 77% of the time. Quoting 0.89 as "priority accuracy" overstates what a human reviewer would call correct. Any external write-up should state the 0.7722 figure alongside it.

Usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "Swift-Support/labse-priority-1.0"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

texts = ["Someone has taken money from my account and nobody is helping me!",
         "How do I activate my new card?"]

with torch.no_grad():
    batch = tok(texts, return_tensors="pt", padding=True, truncation=True, max_length=128)
    probs = model(**batch).logits.softmax(-1)

for text, p in zip(texts, probs):
    print(model.config.id2label[int(p.argmax())], f"{p.max():.3f}", "|", text)

Two things that will silently corrupt results if you get them wrong:

• max_length=128 must match training. It is not stored in the checkpoint.
• Read the label from model.config.id2label, never from a hardcoded index. This checkpoint carries an explicit mapping (0: Low, 1: Medium, 2: High). A wrong index guess does not raise — it silently returns the wrong priority.

Cost: ~1.9 GB resident, roughly 100–300 ms per ticket on CPU. Load the model once at process start, never per request.

Training

base	sentence-transformers/LaBSE
fit portion	train+dev, 49,990 rows
epochs	3 (best epoch: 3 of 3)
learning rate	2e-5
batch size	32
max sequence length	128
class imbalance	class_weight (balanced)
precision	fp16
hardware	Kaggle T4, ~112 rows/s, 22 min wall

best_epoch = 3 of 3 — the model was still improving when training stopped, which is what a consistent labeling target looks like.

Limitations

• Romanized text is synthetic. Singlish is rule-generated from Sinhala and Tanglish is machine-translated, so both are cleaner and more regular than text a human would type. The Singlish and Tanglish numbers above are an optimistic upper bound, and no romanized-specific conclusion from this model should be trusted until it is re-measured on human-typed data.
• Labels are LLM-generated — see the label ceiling section.
• Domain-bound. Derived from BANKING77; behaviour outside retail-banking support is untested.
• Not calibrated. The softmax scores are not probabilities you should threshold on without re-calibrating; a threshold tuned by cross-validation on a sibling task failed to transfer to test in this project.
• Trained and evaluated only on the five tracks listed. LaBSE covers 109 languages, but nothing here measures the other 104.

Citation & provenance

Derived from BANKING77 (PolyAI, CC-BY-4.0), translated into Sinhala and Tamil and romanized into Singlish and Tanglish. Priority labels were generated by an LLM prompt and benchmarked against human annotation as described above.

Training data: Swift-Support/swift-support-tickets-1.0

How to cite

Citing BANKING77 alone is not sufficient attribution for this corpus. BANKING77 supplies the English text and the 77-way intent labels — 20% of the rows and one of the three label columns. The rest is this project's work:

Rows whose text is new (Sinhala, Tamil, Singlish, Tanglish)	52,308 of 65,385 — 80%
Rows inherited from BANKING77 (English)	13,077 — 20%
Label columns inherited	1 (category / intent)
Label columns added	2 (sentiment, priority) — 26,154 new per-ticket assignments
Split	the frozen id-level split is this project's, not BANKING77's

If you use the translated tracks, the sentiment or priority labels, the frozen split, or any of the models, cite this work. If you additionally use the English text or the intent labels, cite BANKING77 as well. Both licences are CC-BY-4.0, so attribution is a licence condition, not a courtesy.

This work — the corpus

@misc{swift_tickets_2026,
  author       = {Sithija Seneviratne and Ruththiragayan Sutharsan and Shazan Shaheed},
  title        = {Swift Support Tickets: A Multilingual Ticket-Triage Corpus for Sinhala,
                  Tamil and English, Including Romanized Code-Mixed Text},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/Swift-Support/swift-support-tickets-1.0}},
  note         = {Derived from BANKING77. Adds four language tracks and the sentiment and
                  priority label columns. Paper in preparation.}
}

This work — the models

@misc{swift_models_2026,
  author       = {Sithija Seneviratne and Ruththiragayan Sutharsan and Shazan Shaheed},
  title        = {Swift: Multilingual Baselines for Support-Ticket Triage in Sinhala,
                  Tamil and English},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Swift-Support}},
  note         = {Intent, sentiment and priority classifiers. Paper in preparation.}
}

BANKING77 — the source of the English text and the intent labels

@inproceedings{casanueva2020banking77,
  author    = {Casanueva, I{\~n}igo and Tem{\v c}inas, Tadas and Gerz, Daniela and
               Henderson, Matthew and Vuli{\'c}, Ivan},
  title     = {Efficient Intent Detection with Dual Sentence Encoders},
  booktitle = {Proceedings of the 2nd Workshop on Natural Language Processing for
               Conversational AI},
  pages     = {38--45},
  year      = {2020},
  publisher = {Association for Computational Linguistics},
  doi       = {10.18653/v1/2020.nlp4convai-1.5},
  url       = {https://aclanthology.org/2020.nlp4convai-1.5/}
}