agentlans/mmBERT-small-spam-detector

mmBERT Multilingual Spam Detector

A multilingual spam detection model fine-tuned on jhu-clsp/mmBERT-small using the agentlans/common-crawl-spam dataset.

• Model Architecture: ModernBERT
• Problem Type: Regression (Single-output spam scoring)
• Number of Labels: 1
• Vocabulary Size: 256,000
• License: MIT

Quick Start & Usage

You can easily use this model for inference with the Hugging Face pipeline:

import torch
from transformers import pipeline

model_name = "agentlans/mmBERT-small-spam-detector"

regressor = pipeline(
    "text-classification",
    model=model_name,
    tokenizer=model_name,
    device=0 if torch.cuda.is_available() else -1,
    function_to_apply="none",
)

messages = [
    "The meeting starts at 10 AM tomorrow.",
    "WIN $$$ NOW!!! FREE MONEY GUARANTEED!!! CLICK HERE!!!",
]

predictions = regressor(messages, batch_size=16, truncation=True)

for message, prediction in zip(messages, predictions):
    score = prediction["score"] # logits for the "spam" label prediction
    print(f"Message: {message}\nPredicted spam score: {score:.3f}\n")

Evaluation & Performance

Test Set Performance

• ROC-AUC: 0.9327 (Evaluated on n = 10,000 test samples from agentlans/common-crawl-spam).
• Decision Rule: In this dataset, a sample is classified as spam if its logit > 0. Points along the ROC curve represent alternative cutoff thresholds for the classifier's output.

Example Predictions & Qualitative Analysis

The dot plot below illustrates predicted spam scores across various categories and languages, highlighting the expected progression from informational messages to pure spam. Note the score variability observed within the three least spam-like categories.

The raw sample classification results are in the following table:

Click here for Table 1

text	language	category	predicted_spam_score
The meeting starts at 10 AM tomorrow.	en	pure_information	-11.5533
La biblioteca cierra a las seis de la tarde.	es	pure_information	-11.8742
Le train pour Paris part à 14 h 30.	fr	pure_information	-5.45476
Die Prüfung findet am Montag statt.	de	pure_information	-10.171
明天的天气预报显示有雨。	zh	pure_information	-7.34299
आज स्कूल सुबह आठ बजे खुलेगा。	hi	pure_information	-3.78344
سيبدأ الاجتماع في الساعة التاسعة صباحًا.	ar	pure_information	-7.88322
A reunião foi transferida para sexta-feira.	pt	pure_information	-10.3215
Your package will arrive tomorrow. Track it on the official website.	en	mostly_informational	-4.51236
Tu factura vence mañana. Puedes consultarla en nuestra aplicación.	es	mostly_informational	2.66382
Votre abonnement sera renouvelé le mois prochain.	fr	mostly_informational	-4.89603
Ihr Kontoauszug ist jetzt online verfügbar.	de	mostly_informational	-1.92531
Our store is open until 9 PM. Check out our new collection!	en	mixed_information_promotion	-5.87974
¡Tu pedido está listo! Aprovecha un 10% de descuento en tu próxima compra.	es	mixed_information_promotion	3.82188
Votre livraison est arrivée. Découvrez également nos offres de la semaine.	fr	mixed_information_promotion	-4.02641
课程已更新，欢迎参加我们的高级付费培训。	zh	mixed_information_promotion	-1.41289
Limited-time offer! Subscribe today to unlock exclusive benefits.	en	suspicious_promotion	4.41937
¡Oferta exclusiva! Compra ahora antes de que termine la promoción.	es	suspicious_promotion	5.47664
Gagnez des cadeaux en vous inscrivant dès aujourd'hui !	fr	suspicious_promotion	4.5007
Herzlichen Glückwunsch! Sichern Sie sich jetzt Ihren Sonderrabatt!	de	suspicious_promotion	2.1179
आपके लिए खास ऑफर! अभी खरीदें और बड़ी बचत करें。	hi	suspicious_promotion	5.47138
CONGRATULATIONS!!! You have won a FREE iPhone. Click now to claim!	en	strong_spam	5.92358
¡FELICIDADES! Has ganado un premio. Haz clic aquí para reclamarlo YA.	es	strong_spam	5.3937
URGENT ! Vous avez gagné 10 000 €. Cliquez immédiatement pour recevoir votre prix !	fr	strong_spam	5.50725
Sie haben einen Geldpreis gewonnen! Klicken Sie sofort auf diesen Link!	de	strong_spam	6.06017
مبروك! ربحت جائزة نقدية ضخمة. اضغط هنا لاستلامها فورًا!	ar	strong_spam	5.99319
WIN $$$ NOW!!! FREE MONEY GUARANTEED!!! CLICK HERE!!! BUY NOW!!!	en	pure_spam	6.16083
¡¡¡GANA DINERO FÁCIL!!! ¡¡¡PREMIO GARANTIZADO!!! ¡¡¡HAZ CLIC YA!!!	es	pure_spam	5.90183
ARGENT GRATUIT !!! GAIN GARANTI !!! CLIQUEZ ICI MAINTENANT !!!	fr	pure_spam	5.54298
KOSTENLOS GELD VERDIENEN!!! 100% GARANTIERT!!! JETZT KLICKEN!!!	de	pure_spam	5.97362
免费送彩金！！！保证中奖！！！立即点击领取！！！	zh	pure_spam	5.99609
बिना मेहनत के लाखों कमाएँ!!! अभी क्लिक करें!!! इनाम पक्का!!!	hi	pure_spam	6.00728
اربح أموالًا مجانية الآن!!! جائزة مضمونة!!! اضغط فورًا!!!	ar	pure_spam	6.23987

Training Details

Click here for training details

Hyperparameters

• Learning Rate: 5e-05

• Train Batch Size: 8

• Eval Batch Size: 8

• Optimizer: AdamW (Fused)

• Number of Epochs: 3.0

• Mixed Precision: BF16

• Gradient Accumulation Steps: 1

• Learning Rate Scheduler: Linear

• Warmup Steps: 0

• Warmup Ratio: 0.0

• Weight Decay: 0.0

• Max Gradient Norm: 1.0

• Number of GPUs: 1

• Seed: 42

Training Results & Evaluation Metrics

Metric	Value
Train Loss	2.8603
Validation Loss	3.9000
Validation F1 Score	N/A (Regression setup)
Total FLOPs	$7.8037 \times 10^{15}$

Speed Performance

• Training Runtime: 1,302.34 seconds
• Train Samples per Second: 184.28
• Evaluation Runtime: 10.82 seconds
• Eval Samples per Second: 924.47

Step	Epoch	Learning Rate	Training Loss	Validation Loss	Validation F1
500	0.05	4.9168e-05	7.3389	N/A	N/A
1000	0.1	4.8335e-05	6.0772	N/A	N/A
1500	0.15	4.7502e-05	5.6355	N/A	N/A
2000	0.2	4.6668e-05	5.4067	N/A	N/A
2500	0.25	4.5835e-05	5.2273	N/A	N/A
3000	0.3	4.5002e-05	4.9198	N/A	N/A
3500	0.35	4.4168e-05	5.2285	N/A	N/A
4000	0.4	4.3335e-05	5.0132	N/A	N/A
4500	0.45	4.2502e-05	4.8953	N/A	N/A
5000	0.5	4.1668e-05	4.5493	N/A	N/A
5500	0.55	4.0835e-05	4.5720	N/A	N/A
6000	0.6	4.0002e-05	4.5171	N/A	N/A
6500	0.65	3.9168e-05	4.6098	N/A	N/A
7000	0.7	3.8335e-05	4.5045	N/A	N/A
7500	0.75	3.7502e-05	4.6530	N/A	N/A
8000	0.8	3.6668e-05	4.2827	N/A	N/A
8500	0.85	3.5835e-05	4.3984	N/A	N/A
9000	0.9	3.5002e-05	4.4387	N/A	N/A
9500	0.95	3.4168e-05	4.1526	N/A	N/A
10000	1.0	3.3335e-05	4.5002	4.2595	N/A
10500	1.05	3.2502e-05	2.7414	N/A	N/A
11000	1.1	3.1668e-05	2.7818	N/A	N/A
11500	1.15	3.0835e-05	2.6185	N/A	N/A
12000	1.2	3.0002e-05	2.7448	N/A	N/A
12500	1.25	2.9168e-05	2.4649	N/A	N/A
13000	1.3	2.8335e-05	2.6484	N/A	N/A
13500	1.35	2.7502e-05	2.6997	N/A	N/A
14000	1.4	2.6668e-05	2.4067	N/A	N/A
14500	1.45	2.5835e-05	2.7081	N/A	N/A
15000	1.5	2.5002e-05	2.5956	N/A	N/A
15500	1.55	2.4168e-05	2.5097	N/A	N/A
16000	1.6	2.3335e-05	2.5163	N/A	N/A
16500	1.65	2.2502e-05	2.6180	N/A	N/A
17000	1.7	2.1668e-05	2.5386	N/A	N/A
17500	1.75	2.0835e-05	2.4471	N/A	N/A
18000	1.8	2.0002e-05	2.5227	N/A	N/A
18500	1.85	1.9168e-05	2.4538	N/A	N/A
19000	1.9	1.8335e-05	2.3036	N/A	N/A
19500	1.95	1.7502e-05	2.2903	N/A	N/A
20000	2.0	1.6668e-05	2.5138	3.9579	N/A
20500	2.05	1.5835e-05	1.1239	N/A	N/A
21000	2.1	1.5002e-05	1.1748	N/A	N/A
21500	2.15	1.4168e-05	1.0715	N/A	N/A
22000	2.2	1.3335e-05	1.1432	N/A	N/A
22500	2.25	1.2502e-05	1.1180	N/A	N/A
23000	2.3	1.1668e-05	1.1273	N/A	N/A
23500	2.35	1.0835e-05	1.1292	N/A	N/A
24000	2.4	1.0002e-05	1.1070	N/A	N/A
24500	2.45	9.1683e-06	1.0862	N/A	N/A
25000	2.5	8.3350e-06	1.0994	N/A	N/A
25500	2.55	7.5017e-06	1.1265	N/A	N/A
26000	2.6	6.6683e-06	1.0964	N/A	N/A
26500	2.65	5.8350e-06	1.0535	N/A	N/A
27000	2.7	5.0017e-06	1.0348	N/A	N/A
27500	2.75	4.1683e-06	1.0017	N/A	N/A
28000	2.8	3.3350e-06	1.0187	N/A	N/A
28500	2.85	2.5017e-06	1.0135	N/A	N/A
29000	2.9	1.6683e-06	0.9879	N/A	N/A
29500	2.95	8.3500e-07	1.0514	N/A	N/A
30000	3.0	1.6667e-09	1.0064	3.9000	N/A

Framework Versions

• Transformers: 5.17.0
• PyTorch: 2.13.0+cu130

Limitations & Bias

• Inherits biases and distributions present in the training dataset, which primarily focuses on webpage spam and features a strong English-language skew.
• May not generalize across all niche or emerging spam vectors.
• Users should carefully evaluate the downstream impacts of false positives and false negatives for their specific deployment use case and adjust classification thresholds accordingly.
• While ModernBERT offers efficiency gains over standard encoder architectures, transformer-based text scoring can have higher latency compared to lightweight linear/bag-of-words baselines.

License

This model is licensed under the MIT License