danjai/scamlens-qs-google-bert-v2

ScamLens quick_score: google-bert-v2

A classifier that reads the wording of an SMS or chat message and gives the probability that it is a scam. It is the quick_score tool of ScamLens, a scam-investigation agent built for a university course project (IS469).

This is one of the candidate models that was trained and compared. It is not the one ScamLens runs with.

Training

Setting	Value
base_model	google-bert/bert-base-uncased
epochs	4
lr	3e-05
batch_size	12
max_length	192
warmup_ratio	0.1
weight_decay	0.01
scheduler	linear
class_weights	balanced
val_ratio	0.15
seed	42

Data

• Training examples: 2171 (1846 to train, 325 to validate)
• Training data SHA-256: 5ea093bb3f3be135ba322f3ea70a96497914f8153b3da19e7fde6ae6512bd12e
• Test data SHA-256: 7ffae3e4dad2531b825b5e9ab44278eb3a9e5d09a3e6b4c140ea32bf21e8be23

The training data is Singapore-style scam and genuine messages: messages taken from or rebuilt from public scam advisories, synthetic messages, generated bank alerts in genuine and scam pairs, and everyday SMS from public corpora.

Results

• Validation F1: 0.990
• Test set: 178 hand-written Singapore messages that are never in the training data. A message counts as a scam at a probability of 0.50 or more. Ranges are 95% confidence intervals.
  • Accuracy: 0.904 (0.852 to 0.940)
  • Scam recall: 0.942 (0.880 to 0.973)
  • Scam precision: 0.899 (0.828 to 0.943)
  • F1: 0.920
  • Genuine messages called a scam: 11 of 74
  • Scams missed: 6 of 104

How to use

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "danjai/scamlens-qs-google-bert-v2"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

text = "DBS: Your account is suspended. Verify at dbs-sg-unlock.com"
inputs = tokenizer(text, return_tensors="pt", truncation=True)
with torch.no_grad():
    scam_prob = torch.softmax(model(**inputs).logits, dim=-1)[0, 1].item()
print(scam_prob)

Labels: 0 = genuine, 1 = scam.

In ScamLens

Add this line to .env. The model downloads the first time quick_score runs.

SCAMLENS_QUICK_SCORE_MODEL=danjai/scamlens-qs-google-bert-v2

ScamLens reads the probability as: high from 0.70, medium from 0.50, low below that.

Limitations

• It judges wording only. It does not check links, phone numbers or senders. In ScamLens those checks are separate tools, and this score is only a first opinion.
• It was trained on English messages in a Singapore setting (local banks, agencies and payment services). Expect it to be less reliable elsewhere.
• The test set is small (178 messages), so the confidence intervals are wide.
• Don't use it alone to block or accuse anyone.