OussamaBenSlama/e5-small-dialect2MSA

E5-Small Dialect2MSA

Overview

Arabic laws, regulations, and other formal documents are usually written in Modern Standard Arabic (MSA). However, users often ask questions in their local dialect.

This difference creates a challenge for semantic retrieval. An embedding model may retrieve the correct MSA passage for an MSA query but perform worse when the same query is written in a dialect.

e5-small-dialect2MSA addresses this problem. It is a fine-tuned version of multilingual-e5-small, trained to improve the retrieval of MSA passages from queries written in MSA or Arabic dialects.

The model supports Saudi, Syrian, Egyptian, Sudanese, and Tunisian Arabic.

The main objective is to retrieve relevant MSA content directly from dialect queries without requiring query rewriting or normalization.

Project Links

• Model: OussamaBenSlama/e5-small-dialect2MSA
• GitHub repository (scripts): Project Scripts
• Training and validation data: Hugging Face Dataset
• Test data: Hugging Face Dataset

Evaluation Results

We compare the base model with the fine-tuned model on 1,238 queries per language. The evaluation measures whether relevant MSA content appears in the top five retrieved results and how highly it ranks.

The results show improvements for MSA and all five supported dialects.

MRR@5

Mean Reciprocal Rank at 5 (MRR@5) measures the ranking position of the first relevant passage in the top five results. Higher values indicate better ranking quality.

Query language	Base model	Fine-tuned model	Improvement (pp)
MSA	0.8411	0.8830	+4.19
Saudi	0.7451	0.8563	+11.12
Syrian	0.7460	0.8590	+11.30
Egyptian	0.7531	0.8556	+10.25
Sudanese	0.7241	0.8485	+12.44
Tunisian	0.6542	0.8265	+17.23

Recall@5

Recall at 5 measures the proportion of queries for which at least one relevant passage appears in the top five results. Higher values indicate better retrieval coverage.

Query language	Base model	Fine-tuned model	Improvement (pp)
MSA	0.9596	0.9830	+2.34
Saudi	0.9273	0.9798	+5.25
Syrian	0.9233	0.9790	+5.57
Egyptian	0.9346	0.9774	+4.28
Sudanese	0.9184	0.9750	+5.66
Tunisian	0.8708	0.9596	+8.88

Results Analysis

1. Better retrieval from Arabic dialects

The fine-tuned model improves both metrics for every supported dialect. This shows that it is better at matching dialect queries with relevant MSA passages.

2. The largest gains occur with Tunisian Arabic

Tunisian queries show the largest improvement in both metrics:

• MRR@5 increases from 0.6542 to 0.8265.
• Recall@5 increases from 0.8708 to 0.9596.

This is particularly important because Tunisian Arabic had the lowest baseline scores.

3. The gap between MSA and dialect retrieval becomes smaller

The base model performs better on MSA queries than on dialect queries. Fine-tuning improves the scores for all dialects and reduces this performance gap.

4. MSA retrieval also improves

The fine-tuned model improves MRR@5 from 0.8411 to 0.8830 and Recall@5 from 0.9596 to 0.9830 for MSA queries.

The fine-tuning therefore improves retrieval for both the original query language and its dialect variants.

These results apply to the current evaluation dataset and setup. Performance on other corpora or real-world queries may differ.

Use Case: Legal AI and Arabic Legal Search

A legal AI system often stores laws and regulations in MSA, while users may ask questions in their local dialect.

For example, a user may ask a legal question in Tunisian Arabic, while the relevant law is written in formal MSA.

With a standard embedding model, the system may need to rewrite the question into MSA before retrieval. This adds an extra step and may change the meaning of the question.

With OussamaBenSlama/e5-small-dialect2MSA, the system can encode the original dialect query and retrieve relevant MSA passages directly.

This approach reduces the need for query rewriting and focuses on direct dialect-to-MSA retrieval.

Model Details

Property	Description
Model	OussamaBenSlama/e5-small-dialect2MSA
Base model	intfloat/multilingual-e5-small
Task	Semantic retrieval
Query languages	MSA, Saudi, Syrian, Egyptian, Sudanese, Tunisian
Target content	MSA text chunks
Training method	Contrastive fine-tuning with hard negatives

Training Data

The training data starts with original MSA queries. Each query is translated into five dialect variants.

The dataset therefore contains queries in six forms:

Text column	Language
query	MSA
saudi_variant	Saudi
syrian_variant	Syrian
egyptian_variant	Egyptian
sudanese_variant	Sudanese
tunisian_variant	Tunisian

Translation Pipeline

We use a two-stage translation pipeline to generate dialect variants:

1. MSA to English: facebook/nllb-200-1.3B
2. English to Arabic dialect: oddadmix/Jisr-MT-50M-AllDialects

English is used as an intermediate language between MSA and the target dialect.

This approach allows us to generate dialect variants using relatively small translation models and limited computing resources.

Training Examples

Each training example contains:

• Query: An MSA query or one of its dialect variants.
• Positive passage: The relevant MSA text chunk.
• Hard negatives: Nine MSA text chunks that are not relevant to the query but are selected as difficult alternatives.

The model learns to place a query closer to its positive passage than to the hard negatives.

This training setup aims to improve the similarity between dialect queries and their corresponding MSA content.

Data Sources

The project uses the following datasets as data sources:

• Arabic Triplet With Multi-Negatives
• Mr. TyDi

The cleaned and translated training, validation, and test data is available in the Hugging Face dataset repositories linked at the top of this README.

Usage

Install the required library:

pip install sentence-transformers

Load the model and compare a dialect query with an MSA passage:

from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

model = SentenceTransformer(
    "OussamaBenSlama/e5-small-dialect2MSA"
)

query = "كيفاش نجم نعمل شكاية؟"  # Tunisian Arabic
passage = "تحدد القوانين الإجراءات اللازمة لتقديم الشكاوى."

query_embedding = model.encode(
    f"query: {query}",
    normalize_embeddings=True
)

passage_embedding = model.encode(
    f"passage: {passage}",
    normalize_embeddings=True
)

score = cos_sim(query_embedding, passage_embedding)

print(score.item())

Limitations

1. Short Queries

The original queries have an average length of approximately 13 tokens.

Short queries may not cover the range of questions found in real applications, including long questions, complex wording, and questions with several parts.

2. Translation Quality

The dialect variants are generated through machine translation:

MSA → English → Arabic dialect.

Translation errors may change the meaning of a query or produce unnatural dialect expressions. This can affect the quality of the training data and the final model.

The translation models were selected partly because of limited computing resources. More powerful translation models or human-written dialect queries may improve the data quality and retrieval performance.

3. Limited Dialect Coverage

The model supports five dialects: Saudi, Syrian, Egyptian, Sudanese, and Tunisian.

Performance on other Arabic dialects, mixed dialects, or other language varieties has not been established by the current evaluation.

Future Work

Potential improvements include:

• Add support for more Arabic dialects.
• Include human-written dialect queries from native speakers.
• Improve translation quality.
• Evaluate on longer and more diverse queries.
• Test retrieval on larger corpora and real-world legal questions.

References

• Fine-tuned model: OussamaBenSlama/e5-small-dialect2MSA
• Base model: intfloat/multilingual-e5-small
• Arabic Triplet With Multi-Negatives: NAMAA-Space dataset
• Mr. TyDi: Castorini dataset
• MSA-to-English translation: facebook/nllb-200-1.3B
• English-to-dialect translation: oddadmix/Jisr-MT-50M-AllDialects