CXBVSWEDRF/electra-small-sms-threat-20260519

ELECTRA Small SMS Threat Classifier

Compact binary SMS spam/threat classifier fine-tuned from google/electra-small-discriminator for an autonomous ML Intern end-to-end experiment on SMS threat detection / smishing classification.

Intended use

Classifies short English SMS-like messages as:

• HAM (0): benign/non-threat message
• THREAT (1): spam/phishing/smishing-like message

This is a research/demo model, not a production anti-fraud system. Use with human review and additional telemetry for real deployments.

Dataset provenance

Primary training/evaluation data: codesignal/sms-spam-collection, a CC-BY-4.0 mirror of the SMS Spam Collection v1. The script deduplicates exact (text,label) pairs and creates a stratified 70/10/20 train/validation/test split with seed 42.

The experiment audited additional SMS/smishing datasets, including ucirvine/sms_spam, electricsheepafrica/africa-smishing-sms-phishing, VusalShahbazovAz/smishing-AZ-SC, shariul-islam/bengali-sms-smishing-dataset, MOZNLP/MOZ-Smishing, dbarbedillo/SMS_Spam_Multilingual_Collection_Dataset, Ngadou/Spam_SMS, CloveAI/india-spam-sms, daviiiidcpp1/sms-spam-combined, and Mustafaege/sms-spam-balanced. The final uploaded checkpoint uses only the real English SMS dataset, because it already met target and avoids synthetic test inflation.

Training recipe

Inspired by BERT-family phishing detection recipes from recent literature: pretrained encoder, AdamW, learning rate 2e-5, batch size 16, multiple epochs, and accuracy/precision/recall/F1 evaluation. ELECTRA-small was selected as a compact encoder with strong speed/performance tradeoff.

Exact local fallback command:

HF_ENDPOINT=https://huggingface.co CUDA_VISIBLE_DEVICES=0 python sms_threat_experiment/train_sms_threat.py   --output_dir sms_threat_experiment/outputs/electra_real_run1   --epochs 5 --batch_size 16 --learning_rate 2e-5 --max_length 160

HF Jobs attempt failed with 401 Unauthorized, so the run used one local NVIDIA L20 GPU with CUDA_VISIBLE_DEVICES=0.

Metrics

Held-out real SMS test split (1,032 rows; 904 ham / 128 threat):

Metric	Value
Accuracy	0.9884
Macro precision	0.9795
Macro recall	0.9665
Macro-F1	0.9729
Positive/threat precision	0.9677
Positive/threat recall	0.9375
Positive/threat F1	0.9524

Baselines on the same test split:

Baseline	Macro-F1	Positive F1
Majority class	0.4669	0.0000
TF-IDF 1-2gram + balanced LogisticRegression	0.9560	0.9231

Target macro-F1 was >= 0.93; achieved 0.9729.

Limitations

• Evaluated on classic English SMS spam data; modern smishing campaigns, non-English SMS, and region-specific scams may differ.
• Positive class is spam/threat-like, not exclusively verified credential-phishing smishing.
• Dataset is small and imbalanced; recall for rare novel threats should be validated before operational use.
• Model card intentionally excludes secrets/tokens and does not include raw sensitive user data beyond public dataset provenance.

Artifacts

• train_sms_threat.py: reproducible training/evaluation script
• experiment_config.md: config summary
• metrics.json: full metrics, split counts, and baseline scores

Generated: 2026-05-19T11:14:47.165059Z

Generated by ML Intern

This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

• Try ML Intern: https://smolagents-ml-intern.hf.space
• Source code: https://github.com/huggingface/ml-intern

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = 'CXBVSWEDRF/electra-small-sms-threat-20260519'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

For non-causal architectures, replace AutoModelForCausalLM with the appropriate AutoModel class.