MahdiAlikhah/laya-persian-sentiment

Laya Persian Sentiment Analysis

A four-class Persian (Farsi) sentiment classification model fine-tuned from convaiinnovations/laya-multilingual, an mmBERT-base model with approximately 322 million parameters.

The model was fine-tuned using the laya training API. Its prediction confidence is temperature-scaled using a held-out calibration set. Calibration quality may vary across datasets and input types.

This is the first release, developed through several rounds of experimentation with training data, label definitions, and training settings.

Highlights

• Four sentiment classes: positive, negative, mixed, and neutral.
• A dedicated mixed class for texts expressing both favorable and unfavorable evaluations.
• Confidence scores with temperature scaling. On the development set, the reported expected calibration error (ECE) was 0.043.
• Designed for informal and formal Persian, including negation, hedged opinions, and contrastive expressions such as «ولی» and «اما».

These capabilities are intended use cases, not a guarantee that every instance of sarcasm, negation, or implicit meaning will be classified correctly.

How to use

The model uses the laya training/inference API and the same question definition used during training.

import os
os.environ["USE_TF"] = "0"

# pip install "laya>=0.4.2" "transformers>=4.48.0" huggingface_hub

import laya
from huggingface_hub import snapshot_download

REPO_ID = "MahdiAlikhah/laya-persian-sentimen"

path = snapshot_download(REPO_ID)  # Authenticate if the repo is private.

import torch
device = "cuda" if torch.cuda.is_available() else "cpu"

agent = laya.load(path, device=device)

QUESTION = {
    "sentiment": {
        "type": "choice",
        "instructions": (
            "Classify the overall sentiment of this Persian text."
        ),
        "criteria": {
            "positive": (
                "The overall evaluation is favorable: satisfaction, "
                "approval, praise, or recommendation."
            ),
            "negative": (
                "The overall evaluation is unfavorable: dissatisfaction, "
                "complaint, or disapproval."
            ),
            "mixed": (
                "Both a genuinely favorable and a genuinely unfavorable "
                "evaluation are expressed."
            ),
            "neutral": (
                "No clear favorable or unfavorable evaluation is expressed, "
                "such as in questions or plain factual statements."
            ),
        },
    }
}

texts = [
    "گوشی عالیه، پشیمون نمیشید",
    "قیمتش خوبه ولی دوربینش ضعیفه",
    "این گوشی چند گیگ رم داره؟",
]

results = agent.predict_batch(texts, QUESTION, batch_size=32)

for text, result in zip(texts, results):
    answer = result["answers"]["sentiment"]
    print(
        "Text:", text,
        "| Label:", answer["choice"],
        "| Confidence:", answer["answer_confidence"],
    )

Use answer_confidence as the confidence score associated with the selected class. If the installed laya version exposes probabilities, those scores can be inspected for the individual classes as well.

Confidence is not a guarantee of correctness. Validate confidence thresholds on your own data before using them for automated decisions.

Label definitions

• positive: The overall evaluation is favorable.
• negative: The overall evaluation is unfavorable.
• mixed: At least one genuinely favorable and one genuinely unfavorable evaluation are expressed. The two sides do not need to be equally strong.
• neutral: No clear favorable or unfavorable evaluation is expressed, including questions, plain facts, and balanced statements such as «نه خوبه نه بد».

A contrast word alone, such as «ولی» or «اما», does not automatically make a text mixed.

Training

• Base model: convaiinnovations/laya-multilingual.
• Task: One choice question with four sentiment classes.
• Training data: Approximately 4.1k labeled Persian texts covering product reviews, service and delivery feedback, social-media posts, and everyday statements.
• Training: 4 epochs, effective batch size 16 (micro-batch 8 with gradient accumulation 2), maximum sequence length 512 tokens, soft cross-entropy loss, shuffled options each epoch, seed 42.
• Calibration: Temperature scaling fitted on approximately 10% of the training texts held out for calibration.

The reported results below should be interpreted in light of the dataset sizes, annotation procedures, and evaluation methodology used for each set.

Evaluation

The following results were reported for held-out evaluation sets. Confirm that each set was kept separate from training and calibration before interpreting these results as independent evaluation.

Evaluation set	Texts	Accuracy	Macro-F1
General held-out set	395	0.901	0.880
External human-annotated set	165	0.745	0.740
Challenging expressions (irony, hedges, negation, contrast)	53	0.906	0.912
Everyday statements	38	0.947	0.936

Per-class F1

Evaluation set	Positive	Negative	Mixed	Neutral
General held-out set	0.88	0.92	0.93	0.78
External human-annotated set	0.63	0.84	0.82	0.67
Challenging expressions	0.95	0.88	0.91	0.91
Everyday statements	1.00	0.88	0.91	0.96

Development set (392 texts): accuracy 0.878, ECE 0.043, Brier score 0.193.

The challenging-expression and everyday-statement sets are small, so their results are indicative rather than conclusive. The external human-annotated set also shows that performance can be substantially lower on data with different characteristics or annotation conventions.

Intended use and limitations

Best suited to short and medium-length Persian texts, including product and service reviews, delivery feedback, customer feedback, social-media posts, and everyday statements of opinion or feeling.

The model has not been evaluated sufficiently for the following domains:

• News and political text
• Legal, medical, or religious writing
• Literary and poetic language
• Very long documents
• Text that mixes Persian with other languages

Sarcasm, heavily implied meaning, ambiguous sentiment, and disagreement about annotation labels can lead to errors. Human review is recommended when the consequences of a wrong prediction matter.

A confidence threshold around 0.9 may be a starting point for experimentation, not a validated universal cutoff. Choose thresholds using representative data from the intended application.

License

This repository declares Apache-2.0. Confirm that the base model's license and applicable terms permit redistribution of the fine-tuned weights under this license before making the repository public.