Utkarsh64/huggingface_distilbert_model

Mental Health Emotion Detection - Enhanced DistilBERT

This model is a fine-tuned DistilBERT for multi-label emotion classification in mental health applications, detecting 28 different emotions from text input with enhanced architecture and advanced training techniques.

Model Description

• Model Type: Enhanced DistilBERT (Fine-tuned)
• Base Model: distilbert-base-uncased
• Task: Multi-label emotion classification
• Dataset: GoEmotions (balanced and enhanced)
• Languages: English
• Architecture: Enhanced with additional layers, focal loss, and class balancing

Performance

Metric	Score
F1-Score	0.298
Precision	0.459
Recall	0.260
Accuracy	89.5%
Improvement	7.6x over baseline

Emotions Detected

The model can detect 28 emotions: admiration, amusement, anger, annoyance, approval, caring, confusion, curiosity, desire, disappointment, disapproval, disgust, embarrassment, excitement, fear, gratitude, grief, joy, love, nervousness, optimism, pride, realization, relief, remorse, sadness, surprise, neutral.

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("YOUR_USERNAME/mental-health-enhanced-distilbert")
model = AutoModelForSequenceClassification.from_pretrained("YOUR_USERNAME/mental-health-enhanced-distilbert")

# Example usage
text = "I'm feeling really anxious about tomorrow"
inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True, max_length=128)

with torch.no_grad():
    outputs = model(**inputs)
    predictions = torch.sigmoid(outputs.logits)

# Get emotion labels
emotions = []
for i, score in enumerate(predictions[0]):
    if score > 0.4:  # Threshold
        emotion = model.config.id2label[i]
        emotions.append((emotion, score.item()))

print(emotions)

Training Details

Enhanced Architecture

• Base: DistilBERT with additional hidden layers
• Enhancements:
  • Layer normalization
  • Dropout regularization
  • Enhanced forward pass with ReLU activations
  • Multi-layer classification head (768 → 512 → 256 → 128 → 28)

Advanced Training Techniques

• Loss Function: Focal Loss for class imbalance handling
• Class Weighting: Advanced weighting for rare emotions
• Data Balancing: Oversampling rare emotions, undersampling common ones
• Optimization: AdamW with cosine scheduling
• Early Stopping: Patience-based with best model saving

Training Data

• Dataset: GoEmotions (balanced subset)
• Training Samples: ~12,750
• Validation Samples: ~2,250
• Preprocessing: Contraction expansion, lowercase normalization
• Balancing: Advanced sampling for 28 emotion categories

Model Architecture

Input Text → DistilBERT Encoder → Enhanced Classification Head
                                        ↓
                                   Hidden Layer 1 (768→512)
                                        ↓
                                   Hidden Layer 2 (512→256)  
                                        ↓
                                   Hidden Layer 3 (256→128)
                                        ↓
                                   Output Layer (128→28)

Intended Use

This model is designed for:

• Mental health chatbots and companions
• Emotion-aware dialogue systems
• Mental health screening tools
• Research in computational psychology
• Empathetic AI applications

Limitations

• Trained primarily on English text
• Performance may vary with very informal language
• Should not be used as sole diagnostic tool for mental health
• Requires context for optimal performance

Training Metrics by Epoch

Epoch	F1-Score	Precision	Recall
1	0.0145	0.0419	0.0089
2	0.1430	0.2797	0.1211
3	0.2141	0.4751	0.1804
4	0.2749	0.4317	0.2340
5	0.2897	0.4524	0.2533
6	0.2981	0.4592	0.2597

Citation

If you use this model, please cite:

@misc{mental-health-emotion-distilbert,
  title={Mental Health Emotion Detection - Enhanced DistilBERT},
  author={Your Name},
  year={2024},
  publisher={Hugging Face},
  url={https://huggingface.co/YOUR_USERNAME/mental-health-enhanced-distilbert}
}

Acknowledgments

• Built on DistilBERT by Hugging Face
• Trained on GoEmotions dataset
• Enhanced with advanced ML techniques for mental health applications