Abdulrhman-Alwehaib/PIRED-Arabic-Retrieval

PIRED

Beats multilingual-e5-base on the Arabic tasks neither model was trained on (48.5 vs 46.7 nDCG@10), at half the size.

PIRED is an Arabic text embedding model for search and retrieval. I built it from scratch: the tokenizer, the pre-training, the contrastive training and the fine-tuning. It has 134M parameters and it only knows Arabic.

You give it a question and a set of passages, it turns each one into a 768-dimensional vector, and the passages closest to the question are the ones most likely to answer it.

Results

PIRED next to multilingual-e5-base (278M parameters) on all 11 Arabic retrieval tasks in MTEB, split into the tasks neither model was trained on and the tasks they saw in training. I ran both models through the same evaluation code (mteb 2.12.10). Each score is the average over the tasks in that group, multiplied by 100. Higher is better, and the better score in each pair is in bold.

Unseen tasks (7)

	PIRED (134M)	multilingual-e5-base (278M)
nDCG@10	48.5	46.7
Recall@100	74.9	73.6

XPQA (product questions), WebFAQ (FAQ pages from the web), PublicHealthQA (health questions), Sadeem (community questions), MKQA (open-domain questions), MultiLongDoc (long documents) and Mintaka (entity questions). Neither model was trained on any of them.

Seen tasks (4)

	PIRED (134M)	multilingual-e5-base (278M)
nDCG@10	67.4	70.1
Recall@100	93.6	94.2

MIRACL, its hard-negative version and Mr.TyDi (Wikipedia search): both models were trained on their train splits. MLQA (questions on Wikipedia): e5-base saw it through xP3, which is part of its training data. PIRED did not. MTEB also lists a second version of the MIRACL hard-negative task, which gives identical scores for Arabic, so it is counted once.

All 11 tasks

	PIRED (134M)	multilingual-e5-base (278M)
nDCG@10	55.4	55.2
Recall@100	81.7	81.1

Note: up to 40% faster than multilingual-e5-base, with half the memory.

How to use it

PIRED comes with its own model code, so load it with trust_remote_code=True:

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

repo = "Abdulrhman-Alwehaib/PIRED-Arabic-Retrieval"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()


def embed(texts, prefix, max_length):
    batch = tokenizer([prefix + t for t in texts], padding=True, truncation=True,
                      max_length=max_length, return_tensors="pt")
    with torch.no_grad():
        hidden = model(**batch).last_hidden_state
    mask = batch["attention_mask"].unsqueeze(-1).to(hidden.dtype)
    vectors = (hidden * mask).sum(1) / mask.sum(1)
    return F.normalize(vectors, dim=-1)


questions = embed(["ما هي عاصمة المملكة العربية السعودية؟"], "استعلام: ", 64)
passages = embed([
    "الرياض هي عاصمة المملكة العربية السعودية وأكبر مدنها.",
    "القاهرة هي عاصمة جمهورية مصر العربية.",
], "نص: ", 256)

print(questions @ passages.T)

The first passage gets the higher score. Questions get the prefix استعلام: and passages get نص: . The model was trained with up to 64 tokens per question and 256 per passage. The tokenizer normalizes Arabic on its own (it drops diacritics and tatweel and unifies the alef forms), so you don't need to clean the text first.

How PIRED was trained

1. Tokenizer. A 64k BPE vocabulary learned only from Arabic text. On 3,000 held-out Arabic passages it uses 1.31 tokens per word, against 1.76 for multilingual-e5-base.
2. Pre-training. Masked language modelling on 10.85B tokens of Arabic web text from FineWeb-2.
3. Contrastive training. 13.0M weak pairs (a headline and its article, a title and its section, a question and its answer, and so on) with in-batch negatives.
4. Fine-tuning. 813k questions, each with 7 hard negatives that a reranker checked. 500k of the questions were written by Qwen3.6-35B-A3B, 247k come from mMARCO (MS MARCO translated to Arabic), and 66k are human-labelled questions from the training splits of MIRACL, Mr.TyDi, TyDi QA and ArabicaQA.

Specs

• Parameters: 134M
• Pre-training data: 10.85B Arabic tokens
• Layers: 12, hidden size 768, 12 attention heads, pre-norm
• Feed-forward: GeGLU, size 2048
• Positions: rotary (RoPE), up to 1024 tokens
• Vocabulary: 64k BPE, learned only from Arabic
• Embedding: 768 dimensions, mean of the last layer over the real tokens, L2-normalized
• Trained lengths: up to 64 tokens per question and 256 per passage

Limitations

• Arabic only. It won't do well on other languages.
• Passages longer than 256 tokens get cut, since that's the longest it saw in training.
• It's built for retrieval. It wasn't tuned for classification, clustering or reranking.

License

CC BY-NC 4.0. Some of the training data (MS MARCO through mMARCO, and XL-Sum) only allows non-commercial use, so PIRED is released for non-commercial use too.