URNOVA

URNOVA is a compact Urdu masked-language model encoder built with the ModernBERT architecture.

It was trained from scratch on an Urdu corpus derived from HPLT 3.0 and is designed for:

Urdu representation learning
masked-language modelling
semantic feature extraction
document and sentence classification
downstream Urdu NLP fine-tuning
research involving longer Urdu sequences

Benchmark results

UrBLiMP zero-shot linguistic evaluation

Zero-shot minimal-pair evaluation on UrBLiMP using left-to-right word pseudo-log-likelihood (pll_word_l2r).

Model	Params	Pairs	Micro Acc.	Macro	Aspect Agreement	Dative Object	Ergativity	Experiencer Subject	Honorific	Noun Phrase	Oblique	Participial Relatives	Subject-Verb Agreement	Word Order
URNOVA-95M	95.2M	5,698	98.17%	97.96%	100.00%	96.23%	99.10%	98.77%	97.39%	99.50%	98.22%	94.68%	96.66%	99.01%
HPLT-BERT Urdu	150.2M	5,698	98.48%	98.14%	99.13%	99.14%	99.28%	97.04%	96.41%	97.75%	98.54%	98.01%	97.62%	98.51%
XLM-R-large	560.1M	5,698	95.75%	94.31%	99.25%	96.23%	96.20%	95.31%	79.74%	93.00%	96.59%	92.36%	96.40%	98.02%
Urdu-RoBERTa-small	126.0M	5,698	94.02%	90.75%	99.75%	88.70%	97.05%	95.80%	70.59%	92.50%	96.23%	89.70%	89.07%	88.12%
XLM-R-base	278.3M	5,698	92.70%	89.32%	99.50%	93.84%	95.36%	92.84%	61.44%	88.50%	96.52%	79.40%	87.79%	98.02%
mmBERT-small	140.9M	5,698	91.08%	88.33%	99.25%	91.78%	95.06%	73.83%	65.36%	87.00%	92.89%	89.70%	89.46%	99.01%

Quick start

Fill-mask inference

from transformers import pipeline

fill_mask = pipeline(
    task="fill-mask",
    model="ProximaAI/urnova-95m",
    token=True,
)

results = fill_mask(
    "پاکستان کی قومی زبان [MASK] ہے۔",
    top_k=5,
)

for result in results:
    print(
        result["token_str"],
        result["score"],
    )

Load the masked-language model

from transformers import (
    AutoModelForMaskedLM,
    AutoTokenizer,
)

model_id = "ProximaAI/urnova-95m"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    token=True,
)

model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    token=True,
)

Extract contextual representations

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "ProximaAI/urnova-95m"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    token=True,
)

encoder = AutoModel.from_pretrained(
    model_id,
    token=True,
)

text = "اردو ایک خوب صورت اور وسیع زبان ہے۔"

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=8192,
)

with torch.inference_mode():
    output = encoder(**inputs)

token_embeddings = output.last_hidden_state
attention_mask = inputs["attention_mask"].unsqueeze(-1)

sentence_embedding = (
    token_embeddings * attention_mask
).sum(dim=1) / attention_mask.sum(dim=1)

Training data and provenance

URNOVA was trained from scratch using an Urdu corpus derived from HPLT 3.0.

Recorded corpus details:

1,921,027 source documents
1,246,973,570 source tokens
approximately 4 billion total training tokens
tokenizer vocabulary size of 50,048
no third-party model initialization

No raw training corpus is distributed in this repository.

See:

Intended use

URNOVA is intended for:

Urdu NLP research
masked-language modelling
encoder-based feature extraction
document classification
sentence classification
semantic similarity research
downstream supervised fine-tuning
experimentation with long Urdu documents

Limitations

The reported benchmark evaluates grammatical minimal-pair preference on UrBLiMP and should not be interpreted as a complete measure of general Urdu language understanding.
Benchmark performance may not generalize to Roman Urdu, code-switched text, regional dialects, or unrelated domains.
Minimal-pair pseudo-log-likelihood evaluation measures sentence preference rather than generative ability, reasoning ability, or downstream task performance.
UrBLiMP results depend on the evaluation and scoring procedure described above and should only be compared with results produced using a compatible methodology.
Long-context capability does not guarantee reliable understanding over every 8,192-token input.
Training on web-derived text may reproduce factual errors, stereotypes, offensive content, or representation gaps found in the source material.

Responsible use

Predictions from URNOVA should not be the sole basis for:

punitive decisions
political profiling
content removal
employment decisions
legal decisions
eligibility decisions
other high-impact automated actions

Applications should include domain-specific validation, human review, and appropriate error analysis.

License and data notice

The original URNOVA model weights, tokenizer artifacts, code, documentation, benchmark reports, and visualizations are made available under the Apache License 2.0, to the extent of the rights held by their respective copyright holders.

The model was trained using an Urdu corpus derived from HPLT 3.0. HPLT licenses its dataset packaging under CC0 but states that it does not own the underlying extracted web text.

Apache-2.0 therefore does not grant rights in underlying training documents or third-party text that may be reproduced by the model.

Third-party benchmark datasets are not covered by URNOVA's Apache-2.0 license. UrBLiMP remains subject to its upstream MIT License and copyright notice.

Downloads last month: 25

Safetensors

Model size

95.2M params

Tensor type

F32

ProximaAI
/

urnova-95m