Saif2409/arabert-hard-sentiment

AraBERT fine-tuned for Arabic hotel-review sentiment

aubmindlab/bert-base-arabertv02 fine-tuned on HARD (Hotel Arabic Reviews Dataset, ~105K Booking.com reviews) for binary sentiment.

model	test accuracy	macro F1	ROC-AUC
TF-IDF + LogisticRegression baseline	87.48%	0.8748	0.9432
this model	91.73%	0.9173	0.9685

33.9% reduction in errors over the baseline (paired bootstrap, p < 0.001).

Important: the labels leak in raw HARD

Booking.com prepends an auto-generated rating descriptor to the review body («مخيب للأمل», «استثنائي», «ضعيف جداً»). It is a deterministic function of the star rating the labels derive from, and it covers 41.3% of the corpus. This model was trained on text with that header stripped. Numbers from work that does not strip it (≈94% for the same baseline) are not comparable.

Accuracy by language register

register	n	accuracy
dialect	1,970	95.84%
msa	1,827	92.50%
unmarked	6,500	90.26%

Dialectal reviews are easier than MSA here, and the gap survives matching on review length within every quintile — dialectal writing in this corpus is blunter and more lexically explicit.

Known limitation

The official ArabertPreprocessor strips emoji, so «الفندق 😍» reaches the model as «الفندق». If emoji carry sentiment in your data, load it with keep_emojis=True and re-tune.

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

repo = "Saif2409/arabert-hard-sentiment"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

text = "الفندق وايد زين والطاقم متعاون"
with torch.inference_mode():
    probs = torch.softmax(model(**tok(text, return_tensors="pt")).logits, -1)[0]
print(model.config.id2label[int(probs.argmax())], float(probs.max()))

Full code, error analysis and probe suite: https://github.com/Saif2409/arabic-sentiment