danjai/scamlens-qs-google-bert-v7

ScamLens quick_score: google-bert-v7

A classifier that reads the wording of an SMS or chat message and gives the probability that it is a scam. It is the quick_score tool of ScamLens, a scam-investigation agent built for a university course project (IS469).

This run did not train well (validation F1 0.719). It is published as a record of what was tried, not for use.

Training

Setting	Value
base_model	google-bert/bert-base-uncased
epochs	2
lr	1e-06
batch_size	8
max_length	192
warmup_ratio	0.1
weight_decay	0.01
scheduler	linear
class_weights	balanced
val_ratio	0.2
seed	42

Data

• Training examples: 1901 (1522 to train, 379 to validate)
• Files: sg_quick_score_train.csv
• Validation split by message template (1387 templates), so near-copies of a training message are never in validation
• Training data SHA-256: a347f8eb27ae4db455f8e5523e0a983f89bd6be2cbc0af93775291fa5de1ed9b
• Test data SHA-256: 7ffae3e4dad2531b825b5e9ab44278eb3a9e5d09a3e6b4c140ea32bf21e8be23

The training data is Singapore-style scam and genuine messages: messages taken from or rebuilt from public scam advisories, synthetic messages, generated bank alerts in genuine and scam pairs, and everyday SMS from public corpora.

Results

• Validation F1: 0.719
• Test set: 178 hand-written Singapore messages that are never in the training data. A message counts as a scam at a probability of 0.50 or more. Ranges are 95% confidence intervals.
  • Accuracy: 0.747 (0.679 to 0.805)
  • Scam recall: 0.827 (0.743 to 0.888)
  • Scam precision: 0.761 (0.675 to 0.830)
  • F1: 0.793
  • Genuine messages called a scam: 27 of 74
  • Scams missed: 18 of 104

How to use

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "danjai/scamlens-qs-google-bert-v7"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

text = "DBS: Your account is suspended. Verify at dbs-sg-unlock.com"
inputs = tokenizer(text, return_tensors="pt", truncation=True)
with torch.no_grad():
    scam_prob = torch.softmax(model(**inputs).logits, dim=-1)[0, 1].item()
print(scam_prob)

Labels: 0 = genuine, 1 = scam.

In ScamLens

Add this line to .env. The model downloads the first time quick_score runs.

SCAMLENS_QUICK_SCORE_MODEL=danjai/scamlens-qs-google-bert-v7

ScamLens reads the probability as: high from 0.70, medium from 0.50, low below that.

Limitations

• It judges wording only. It does not check links, phone numbers or senders. In ScamLens those checks are separate tools, and this score is only a first opinion.
• It was trained on English messages in a Singapore setting (local banks, agencies and payment services). Expect it to be less reliable elsewhere.
• The test set is small (178 messages), so the confidence intervals are wide.
• Don't use it alone to block or accuse anyone.