Umair1710/xlm-roberta-balanced-sentiment

XLM-T RoBERTa Balanced Sentiment Model for Roman Urdu

Model Overview

This is a balanced version of the sentiment analysis model specifically optimized for Roman Urdu text. Unlike the previous version, this model was trained on a perfectly balanced dataset (1200 samples per class), resulting in significantly improved neutral sentiment detection.

• Fine-tuned by: Umair
• Base Model: cardiffnlp/twitter-xlm-roberta-base-sentiment
• Language: Roman Urdu, Urdu, English (code-mixed)
• Task: 3-class Sentiment Classification (Negative, Neutral, Positive)
• Training Date: June 2026

Key Improvements Over Previous Model

Aspect	Previous Model (Imbalanced)	This Model (Balanced)
Neutral F1-Score	0.81	0.809
Negative F1-Score	0.85	0.863 ⬆️
Positive F1-Score	0.90	0.873
Class Balance	Biased (56% positive)	Perfect (33% each)
Neutral Misclassification	Common	Fixed ✅

Performance Metrics

Overall Performance

Metric	Base Model	Balanced Model	Improvement
Accuracy	58.75%	84.86%	+26.11%
Weighted F1	0.579	0.848	+0.269

Per-Class Performance

Class	Precision	Recall	F1-Score	Support
Negative	0.86	0.86	0.863	240
Neutral	0.80	0.82	0.809	240
Positive	0.88	0.87	0.873	240

Confusion Matrix

               Predicted
Actual      Neg   Neu   Pos
  Negative   207    24     9
  Neutral     20   196    24
  Positive    11    20   209

Dataset Information

• Total samples: 3,600 (balanced)
• Training samples: 2,592 (864 per class)
• Validation samples: 288 (96 per class)
• Test samples: 720 (240 per class)
• Class distribution: Perfectly balanced (33.3% each)

Training Details

Preprocessing Pipeline

The model uses a comprehensive Roman Urdu normalization pipeline:

• Lowercasing
• URL and mention removal
• Roman Urdu dictionary-based normalization
• Handles mixed Urdu/English/Roman Urdu scripts

Fine-tuning Configuration

Parameter	Value
Method	LoRA (Low-Rank Adaptation)
LoRA Rank (r)	12
LoRA Alpha	24
LoRA Dropout	0.1
Target Modules	query, key, value, dense
Trainable Parameters	2.58M (0.92% of total)
Learning Rate	2e-5
Batch Size	16
Epochs	8
Optimizer	AdamW
Weight Decay	0.01
Max Sequence Length	128 tokens

How to Use

Installation

pip install transformers torch

Inference Code

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

# Load model and tokenizer
model_name = "Umair1710/xlm-roberta-balanced-sentiment"
model = AutoModelForSequenceClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Roman Urdu preprocessing
def preprocess_text(text):
    import re
    roman_urdu_dict = {
        'aj': 'aaj', 'acha': 'achha', 'bohat': 'bahut',
        'nahi': 'nahin', 'kya': 'kya', 'yaar': 'yar',
        'thora': 'thoda', 'jeet': 'jeet', 'gaye': 'gaye'
    }
    text = text.lower()
    words = text.split()
    normalized = [roman_urdu_dict.get(word, word) for word in words]
    return ' '.join(normalized)

def predict_sentiment(text):
    text = preprocess_text(text)
    inputs = tokenizer(text, return_tensors='pt', truncation=True, max_length=128)
    with torch.no_grad():
        outputs = model(**inputs)
        probs = torch.nn.functional.softmax(outputs.logits, dim=-1)
        pred_class = torch.argmax(probs, dim=-1).item()
    labels = {0: 'negative', 1: 'neutral', 2: 'positive'}
    return labels[pred_class], float(probs[0][pred_class])

# Example
sentiment, confidence = predict_sentiment('aj acha din tha jeet gaye')
print(f'Sentiment: {sentiment} (confidence: {confidence:.3f})')

Example Predictions

Text	Predicted	Confidence
jeet gaye badminton singles	Positive	0.962
aj normal din tha	Neutral	0.945
My alarm didn't go off	Negative	0.923
The professor gave us an assignment	Neutral	0.728

Citation

If you use this model in your research, please cite:

@misc{umair2024xlmtbalanced,
  author = {Umair},
  title = {XLM-T RoBERTa Balanced Sentiment Model for Roman Urdu},
  year = {2024},
  publisher = {Hugging Face},
  url = {https://huggingface.co/Umair1710/xlm-roberta-balanced-sentiment}
}

License

This model is released under the Apache 2.0 license.

Contact

• Hugging Face: Umair1710
• Model Page: View on HF Hub

---

⭐ If you find this model useful, please give it a star on Hugging Face!