Model Card for FaizAhmadDev/sms-spam-detection-distilbert

A DistilBERT model fine-tuned for SMS spam detection.

Model Details

Model Description

This model is a fine-tuned version of distilbert-base-uncased specifically trained to classify SMS messages as either 'ham' (legitimate) or 'spam'. It achieves high accuracy, precision, recall, and F1-score on the SMS Spam Collection Dataset, demonstrating its effectiveness in distinguishing unwanted promotional or malicious messages from regular communication.

• Developed by: FaizAhmadDev
• Model type: DistilBertForSequenceClassification
• Language(s) (NLP): English
• License: Apache 2.0
• Finetuned from model: distilbert-base-uncased

Model Sources

• Repository: https://huggingface.co/FaizAhmadDev/sms-spam-detection-distilbert

Uses

Direct Use

This model can be used directly for classifying new, unseen SMS messages as either 'ham' or 'spam'. It can be integrated into applications requiring automated SMS filtering or moderation.

Out-of-Scope Use

This model is specifically trained for English SMS spam detection. It is not designed for:

• Classifying messages in other languages without further fine-tuning.
• General text classification tasks beyond binary spam/ham detection.
• Detecting image-based or multimedia spam.
• Providing explanations for its classifications (it's a black-box model).

Bias, Risks, and Limitations

The model was trained on the SMS Spam Collection Dataset. While this dataset is widely used, it may contain biases reflecting historical spam patterns and language use. As such, the model's performance may:

• Vary on different SMS message distributions, especially with new or evolving spam tactics.
• Show reduced accuracy on highly ambiguous messages or messages using slang/regionalisms not well-represented in the training data.
• Exhibit biases if certain demographic or linguistic groups' messages are disproportionately labeled as spam or ham in the training data.

Recommendations

Users (both direct and downstream) should be aware of these potential biases and limitations. Regular monitoring of the model's performance on new, real-world data is recommended. Consider retraining with updated datasets to adapt to evolving spam characteristics.

How to Get Started with the Model

Use the code below to load the model and tokenizer from the Hugging Face Hub and make predictions.

from transformers import DistilBertForSequenceClassification, DistilBertTokenizer
import torch

# Load the model and tokenizer
model_name = "FaizAhmadDev/sms-spam-detection-distilbert"
tokenizer = DistilBertTokenizer.from_pretrained(model_name)
model = DistilBertForSequenceClassification.from_pretrained(model_name)

# Set model to evaluation mode and move to device
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model.to(device)
model.eval()

def predict_spam(message):
    inputs = tokenizer(
        message,
        truncation=True,
        padding=True,
        max_length=128,
        return_tensors='pt'
    )
    inputs = {key: val.to(device) for key, val in inputs.items()}

    with torch.no_grad():
        outputs = model(**inputs)
        probabilities = torch.nn.functional.softmax(outputs.logits, dim=-1)
        prediction = torch.argmax(probabilities, dim=-1).item()
        confidence = probabilities[0][prediction].item()

    label = 'SPAM' if prediction == 1 else 'HAM'
    return label, confidence

# Example usage:
message_1 = "Congratulations! You've won a $1000 gift card. Click here to claim now!"
message_2 = "Hey, are we still meeting for lunch tomorrow at 1pm?"

label_1, conf_1 = predict_spam(message_1)
label_2, conf_2 = predict_spam(message_2)

print(f"Message: '{message_1[:50]}...'\nPrediction: {label_1}, Confidence: {conf_1:.2f}")
print(f"Message: '{message_2[:50]}...'\nPrediction: {label_2}, Confidence: {conf_2:.2f}")

Training Details

Training Data

The model was fine-tuned on the SMS Spam Collection Dataset. This dataset contains approximately 5,572 English SMS messages, each labeled as either 'ham' (legitimate) or 'spam'. The dataset exhibits class imbalance, with a significantly larger number of 'ham' messages.

Training Procedure

Preprocessing

1. Column Selection & Renaming: Only 'v1' (label) and 'v2' (message) columns were retained and renamed to 'label' and 'message'.
2. Duplicate Removal: Duplicate SMS messages were removed to ensure data uniqueness.
3. Text Cleaning: Messages were converted to lowercase, extra whitespaces were removed, and messages shorter than 3 characters were filtered out.
4. Label Encoding: Categorical labels ('ham', 'spam') were encoded into numerical format (0 for 'ham', 1 for 'spam').
5. Train-Test Split: The dataset was split into 80% training and 20% testing sets using train_test_split with stratify to maintain class distribution.
6. Tokenization: Messages were tokenized using DistilBertTokenizer, with truncation=True and padding=True, and a max_length of 128 tokens.

Training Hyperparameters

• Training regime: Mixed precision (if CUDA available, typically fp16 by default in Hugging Face Trainer with GPU)
• Number of Epochs: 3
• Per Device Training Batch Size: 16
• Per Device Evaluation Batch Size: 16
• Warmup Steps: 100
• Weight Decay: 0.01
• Optimizer: AdamW (default for Hugging Face Trainer)
• Learning Rate: 5e-5 (default for Hugging Face Trainer)

Evaluation

Testing Data, Factors & Metrics

Testing Data

The model was evaluated on 20% of the SMS Spam Collection Dataset, which comprised 1033 unseen messages.

Metrics

Standard classification metrics were used:

• Accuracy: Overall proportion of correctly classified messages.
• Precision (for 'spam' class): Ability of the model to identify only relevant instances (spam messages).
• Recall (for 'spam' class): Ability of the model to find all relevant instances (all spam messages).
• F1-Score (for 'spam' class): Harmonic mean of precision and recall.
• Confusion Matrix: Provides a detailed breakdown of correct and incorrect classifications.

Results

Summary

After 3 epochs of fine-tuning, the model achieved the following performance on the test set:

Metric	Value
Accuracy	99.23%
Precision (Spam)	0.9767
Recall (Spam)	0.9618
F1-Score (Spam)	0.9692

Confusion Matrix:

• True Negatives (Correct Ham): 899
• False Positives (Ham as Spam): 3
• False Negatives (Spam as Ham): 5
• True Positives (Correct Spam): 126

These results indicate a highly effective model with strong generalization capabilities for SMS spam detection, particularly with a good balance of precision and recall for the positive 'spam' class.

Model Card Contact

[https://huggingface.co/FaizAhmadDev/sms-spam-detection-distilbert]