xlm-roberta-base-pakistani-ner

A named entity recognition model for Pakistani news, court, and Wikipedia text. It is XLM-RoBERTa-base fine-tuned to tag 20 entity types with BIO labels. This release is the first training run (4 epochs).

Entity types

PERSON, CITY, PROVINCE_REGION, COUNTRY, LANDMARK_FACILITY, GOVERNMENT_BODY, POLITICAL_PARTY, ORGANIZATION, EDUCATIONAL_INSTITUTION, COURT_LEGAL_BODY, LAW_ACT, EVENT, MEDIA_OUTLET, SPORTS_TEAM, DATE, TIME, MONEY, PERCENT, PHONE_NUMBER, ID_NUMBER

Tagging is flat (no nested entities) and the longest sensible span wins. Titles such as Mr, Dr, or Justice are not part of PERSON. Case numbers and law-report citations are not ID_NUMBER, and years inside them are not DATE. Durations such as "two weeks" are DATE. A capitalised bare court name such as "High Court" is COURT_LEGAL_BODY, while a generic "the court" is not an entity. PHONE_NUMBER and ID_NUMBER (CNIC) were trained on synthetic examples only.

How to use

from transformers import pipeline

ner = pipeline(
    "token-classification",
    model="your-username/xlm-roberta-base-pakistani-ner",
    aggregation_strategy="first",
)

print(ner("The Supreme Court heard the case in Islamabad on Monday."))

The model was trained with the label on the first word piece of each word, so aggregation_strategy="first" matches how it was trained.

The scores below were produced with the project's own word tokenizer and prediction script in the code repository. The Hugging Face pipeline splits text differently, so results can differ slightly, especially around punctuation, numbers, and possessives.

Training data

• Real text: 15,390 sentences collected from Pakistani news, court, and Wikipedia sources. Roughly 18% comes from openly licensed text (Wikipedia and Wikinews), 24% from public court records (Supreme Court judgments), and 58% from news (headlines, Dawn, and Express Tribune).
• Synthetic text: 5,390 template-generated sentences with exact labels, covering phone numbers, CNICs, money, percent, date and time patterns, and rare entity types. All numbers are random and belong to no real person.
• The final training set has 18,245 sentences: 5,390 synthetic, 9,516 machine-labeled sentences judged reliable, 2,974 machine-labeled sentences whose uncertain spans were masked from the loss, 186 human-verified sentences, and 179 sentences labeled by an AI assistant and not human-verified.
• Machine labels came from a pipeline of regex rules, gazetteers, and spaCy draft predictions, followed by rule-based fixes. Training and test sets were split at the document level, and near-duplicate sentences were removed to prevent leakage.
• Human-verified gold sentences were first drafted by an AI assistant, then every sentence was reviewed by a human in Label Studio.

Training procedure

• Base model: FacebookAI/xlm-roberta-base (MIT license)
• 4 epochs, learning rate 3e-5 with 10% warmup and linear decay, batch size 16, weight decay 0.01, fp16, maximum length 192 word pieces, seed 42
• Only the first word piece of each word carries a label. Other pieces and masked spans are ignored by the loss.
• Human-verified sentences were repeated 3 times and AI-labeled extra sentences 2 times, because they are a small share of the data.
• The epoch with the best validation F1 was kept.
• Hardware: Google Colab, NVIDIA Tesla T4 (16 GB)

Evaluation

Scores are entity-level exact match: an entity counts only if its start, end, and type all match.

Test set	Sentences	Entities	Micro F1
Human-verified, random sample	200	426	0.816
Human-verified, enriched for weaker types	100	338	0.812
Both human-verified sets pooled	300	764	0.814 (precision 0.806, recall 0.822)
Organizers' hidden benchmark	820	n/a	0.938

Per-type results on the human-verified sets:

• Strongest on the random sample (F1): MONEY 0.94, PERSON 0.92, MEDIA_OUTLET 0.91, POLITICAL_PARTY 0.88, PROVINCE_REGION 0.87, COUNTRY 0.86, COURT_LEGAL_BODY 0.86.
• Weakest (F1 on the random sample / on the weaker-type set): LAW_ACT 0.48 / 0.42, ORGANIZATION 0.45 / 0.30, LANDMARK_FACILITY 0.36 / 0.64, GOVERNMENT_BODY 0.71 / 0.84.
• Perfect or near-perfect scores on rare types in the random sample rest on 2 to 5 entities and should be ignored.

PHONE_NUMBER and ID_NUMBER have no real test data. On a synthetic test of 188 sentences (new templates, unseen number formats, and decoy numbers), F1 was 0.990 for PHONE_NUMBER and 0.918 for ID_NUMBER. These numbers are optimistic because the test is synthetic and close to the training templates.

Intended use

Extracting entities from English-language Pakistani news and legal-style text, for research, prototyping, and as a starting point for further fine-tuning.

Out-of-scope use

• Not validated for redacting personal data in production. Phone number and CNIC detection was trained on synthetic data only and has not been checked on real documents.
• Not for identifying or profiling individuals.
• Not tested on domains outside news, court, and Wikipedia text.

Known limitations

• Most training labels are machine-made. Only 186 of the 18,245 training sentences are human-verified. In total, 764 sentences were human-verified across the project, including the test sets and a later batch used only for an experimental model that is not this release.
• An AI assistant drafted the first corrections for the human-verified sentences and labeled the 179 extra training sentences, which were not human-verified.
• Test sets are small. Rare-type scores rest on 2 to 18 entities and are noisy. Differences below about 0.02 F1 are within noise.
• Test sets are not fully unseen. Error categories were studied on them. No test text was added to training and no test label was edited.
• ORGANIZATION, LAW_ACT, and LANDMARK_FACILITY are the weakest types. ORGANIZATION is often confused with GOVERNMENT_BODY and POLITICAL_PARTY.
• Phone and CNIC scores come from synthetic data only.
• Annotation conventions are judgment calls, such as durations as DATE and bare court names as COURT_LEGAL_BODY. Another benchmark may label such cases differently.
• English-first. Urdu was only a zero-shot check: 21 of 27 expected entities were found with the right type on 8 sentences written by the authors and not checked by a native annotator. It is a sanity check, not a benchmark.

License

The license field is set to other. The base model, XLM-RoBERTa-base, is MIT licensed. The training text includes Wikipedia (CC BY-SA 4.0), and news and court-judgment text that was used only for training and evaluation and is not redistributed. Replace other at the top of this file with the license you decide on after checking those source terms.

Author and code

Muhammad Sarim Umer. Code: https://github.com/MuhammadSarimUmer/Fine-Tune-NER-XLMRoBERTa