SavienWalker/bulwark-scam-detector

Bulwark Scam Detector

A multilingual scam / spam / legitimate message classifier, specialised for Turkish. Fine-tuned from Horizon-Labs/phishing-scam-detector-base (mmBERT-base, 307M parameters) to fix one specific weakness: false alarms on real Turkish institutional messages (bank, cargo, e-Devlet, operator, OTP notifications).

Code: github.com/SavienWalker/bulwark · Benchmark: SavienWalker/bulwark-bench-tr

🇹🇷 Türkçe özet: Bulwark; SMS, e-posta ve kısa mesajları dolandırıcılık / spam / temiz olarak ayırır. Gerçek Türk kurum mesajlarında mevcut modelin temiz mesajların yarısına verdiği yanlış alarmı %14'e indirir, istenmeyen mesajları yakalama oranı ise %98'de kalır. Diğer dillerdeki temel yeteneği korunmuştur.

Headline result

On a held-out half of real Turkish SMS from a personal inbox (opened once, after all model choices were made):

	Base model (threshold 0.5)	Bulwark (threshold 0.6)
False alarm on legitimate messages ↓	53% (54/102)	14% (14/102)
Legitimate message labelled fraud ↓	26%	9%
Unwanted (spam + fraud) caught ↑	100% (52/52)	98% (51/52)
3-class accuracy ↑	64%	87%

The base model flags roughly every second real bank / cargo / e-Devlet notification as unwanted because it was trained on synthetic scams plus generic web text, and has rarely seen what a real institutional message looks like. Bulwark keeps the base model's recall while learning the difference.

Labels and usage

id	label	meaning
0	legitimate	genuine personal or institutional message
1	spam	advertising from a real business
2	fraud	tries to steal money, card data, passwords, codes, identity or trust

Recommended decision rule: treat a message as unwanted when p(spam) + p(fraud) > 0.6 (chosen on the development split; stored in bulwark_config.json).

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "SavienWalker/bulwark-scam-detector"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()

text = "PTT: Gonderiniz adres eksikligi nedeniyle bekletilmektedir. 0,99 TL odeme yapin: ptt-gonderitakip.com"
with torch.no_grad():
    p = torch.softmax(model(**tok(text, return_tensors="pt", truncation=True, max_length=256)).logits, -1)[0]
probs = {model.config.id2label[i]: round(v.item(), 3) for i, v in enumerate(p)}
unwanted = probs["spam"] + probs["fraud"] > 0.6
print(probs, "unwanted" if unwanted else "ok")

Evaluation

All numbers compare the base model and Bulwark on the same messages. Sets are described in bulwark-bench-tr. Messages marked ambiguous (mostly illegal-betting ads) are excluded from metrics.

Set	Type	Metric	Base	Bulwark
Real SMS, test half (212)	real, held out	false alarm ↓	53%	14%
		unwanted caught ↑	100%	98%
Real e-mail (40)	real, development set	false alarm ↓	33%	19%
Hard set v2 (60)	synthetic, human-reviewed	fraud caught ↑	82%	95%
		false alarm ↓	25%	0%
		contrast pairs fully correct ↑	9/13	12/13
Hard set v1 (42)	synthetic, human-reviewed	contrast pairs fully correct ↑	5/7	7/7
UCI SMS Spam, held-out half (2,787)	real, English	unwanted caught ↑	87%	96%
Multilingual probe (54; de, fr, es, pt, ru, ar)	synthetic, easy	3-class accuracy	54/54	54/54

Contrast pairs are a fake message and the genuine message it imitates (e.g. fake vs. real HGS toll notice); a pair counts only if both are classified correctly — the clearest test of whether a model reads the signals (domain, request, tone) rather than the topic.

How to read these numbers honestly:

• The only fully clean real-world number is the SMS test half. The e-mail set was used for error analysis, so it is reported as development data.
• Hard sets and training templates were written by the same author (an LLM, Claude). A style overlap could inflate those scores. They are reported separately and labelled synthetic.
• The SMS test half contains no non-ambiguous real fraud — real fraud in a personal inbox is almost all illegal betting. Fraud recall is therefore measured on the (synthetic) hard sets only.
• English: half of the UCI SMS Spam Collection was used as replay data during training, so the English gain is in-distribution; the honest claim is "no regression", not "better than the base model".
• Other languages: a small, easy probe shows no regression. Hard cases outside Turkish were not measured.

Training

• Base: Horizon-Labs/phishing-scam-detector-base (Apache-2.0), itself built on jhu-clsp/mmBERT-base (MIT).
• Data (~9k messages):
  • ~8k Turkish synthetic messages from 190 hand-written templates with slot filling (institution, amount, date, code, look-alike domain). Covers real institutional notifications, real-business ads, link-based phishing, and link-free social engineering. Patterns were drawn from public warnings (KGM, SGK, PTT, EGM) and public complaint narratives; no complaint text was copied.
  • ~1k English messages from the UCI SMS Spam Collection (CC BY 4.0) as replay against forgetting; spam vs. fraud for these was taken from the base model's own prediction (distillation).
• Leakage control: every training message was compared with every evaluation message (token-set Jaccard); anything ≥ 0.5 was removed.
• Personalisation balancing: scammers now use leaked personal data (name, card last 4 digits, plate, old address). These details were injected into legitimate and fraudulent training messages at equal rates, so that "the message knows my name" is not learned as a sign of legitimacy.
• Recipe: word embeddings frozen (110M of 307M parameters trained), bf16, AdamW lr 2e-5, 2 epochs, max length 256, class-weighted loss, fraud examples down-sampled to the legitimate count. Runs on a single 6 GB laptop GPU (RTX 4050) in about 6 minutes.
• Model selection: on a development half of the real SMS, split by sender so the same institution never appears in both halves.

Scope and limitations

• One message at a time. Many real scams unfold over a phone call, a long WhatsApp conversation, a fake website or an ad. Bulwark sees only the text it is given; the first message of a slow scam often looks harmless.
• A message containing your real personal details is not proof that it is genuine. Leaked databases make this trivial for scammers.
• Illegal-betting ads are labelled fraud but treated as ambiguous in evaluation.
• Small real evaluation sets (hundreds, not thousands). Treat the numbers as indicative.
• Not a replacement for a bank's or carrier's own fraud systems. Do not use it to make decisions about people.

License and attribution

Apache-2.0. Derived from Horizon-Labs/phishing-scam-detector-base (Apache-2.0) and jhu-clsp/mmBERT-base (MIT). English replay data: Almeida, T.A. & Gómez Hidalgo, J.M., SMS Spam Collection (UCI, CC BY 4.0).