Gillone06/Distilled_Unified_Model_Cardiff_Deberta

Model Card for Model ID

Architecture: Hybrid knowledge distillation system combining:

🧠 Base Model: DeBERTa-v3-base (student model)
👨🏫 Teacher Model: CardiffNLP/twitter-roberta-base-sentiment-latest
⚙️ Custom Components:
    Hidden state projection layers (12 transformer layers)
    Attention matrix alignment module
    Class-weighted focal loss

Model Description

This DeBERTa-v3-base model is fine-tuned for ternary sentiment classification (negative/neutral/positive) using multi-domain data. It employs three-level knowledge distillation:

Logit Distillation: KL divergence loss between student/teacher predictions
Full-Layer Alignment: Layer-wise hidden state projection with cosine similarity
Attention Transfer: Last 4 layers' attention pattern matching

Key Features:

Neutral class upweighting (10x multiplier)
Gradient checkpointing + FP16 optimization
Combined focal loss (γ=3) + distillation

Uses

Direct Use

from transformers import AutoTokenizer, AutoModelForSequenceClassification  

tokenizer = AutoTokenizer.from_pretrained("Gillone06/Distilled_Unified_Model_Cardiff_Deberta")  
model = AutoModelForSequenceClassification.from_pretrained("Gillone06/Distilled_Unified_Model_Cardiff_Deberta")  

text = "Product was great but delivery took weeks!"  
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)  
outputs = model(**inputs)  
prediction = ["negative", "neutral", "positive"][outputs.logits.argmax().item()]  

Out-of-Scope Use

Medical/legal document analysis
Sarcasm detection
Multi-label classification

Training Details

Training Data

Combined dataset from:

Social Media: tweet_eval, sentiment140
Reviews: IMDB, Yelp, Amazon Customer Reviews
Multi-Domain: Custom parsed reviews (balanced subset)

Class Distribution:

Negative: 38.2%
Neutral: 12.1%
Positive: 49.7%

Training Hyperparameters

Parameter:
-Learning Rate	5e-6
-Batch Size	20 (effective)
-Gradient Accumulation	4 steps
-Warmup Ratio	10%
-Max Seq Length	512
-Dropout	0.4
-Epochs	1+ (early stop)
-Optimizer	AdamW
-LR Scheduler	Cosine w/ restarts

Evaluation

Metric(F1-Score):Negative(84.1)---Neutral(73.4)---Positive(89.7)---Macro Avg(82.4)

Accuracy:-----------------------------------------------------------Macro Avg(86.2%)

Key Findings:

15% relative improvement on neutral class vs baseline
3.2% higher cross-domain accuracy than teacher model

Technical Specifications

Model Architecture and Objective

Knowledge Transfer Mechanics:

Hidden States: Layer-wise projection + LN normalization
Attention: Last 4 layers with GELU-projection
Temperature: Annealed from 2.0 → 0.5 during training

Limitations

Performance degrades on texts with mixed sentiments
Requires ≥5 words for reliable classification
English-only limitation