DanishMahdi/avey-b-snd

Avey-B Sindhi 25M

A compact Sindhi encoder pretrained from scratch with masked-language modeling.

This is an encoder-only model, not a chat model or free-text generator. It is intended for fine-tuning on Sindhi classification and token-labeling tasks, representation extraction, fill-mask experiments, and continued pretraining.

Results

Evaluation	Result
Held-out Sindhi MLM loss	2.9448
Held-out Sindhi MLM perplexity	19.01
Masked-token top-1 / top-5 accuracy	46.38% / 67.39%
WikiANN Sindhi entity F1	41.59% ± 2.68%
Batch-1, length-128 latency	1.78 ms
Batch-16, length-512 throughput	401,931 tokens/s
Peak VRAM at batch 16, length 512	148 MiB
Sindhi tokenizer fertility	4.04 characters/token

The MLM evaluation uses 15,351 deterministically selected masked tokens from the held-out HPLT validation split. WikiANN is especially small for Sindhi: only 100 train, 100 validation, and 100 test examples. NER reports mean ± sample standard deviation over 5 seeds after selecting from learning rates 2e-05, 6e-05, 0.0001, 0.0005 by mean validation F1. Each run uses 20 epochs, batch size 16, and maximum sequence length 128. Treat the NER number as a low-resource indicator rather than a broad estimate of Sindhi capability.

Sindhi WikiANN comparison

Entity-level F1 is the primary NER metric. Token accuracy is intentionally not reported for Avey-B because the frequent non-entity (O) label can make accuracy look high even when entity recognition is weak.

The closest published results found on the same Sindhi WikiANN target split are from Hangya et al. (EMNLP 2022). Their models were trained on English NER and transferred to Sindhi with the encoder frozen; Avey-B was instead fine-tuned end-to-end on WikiANN's 100 Sindhi training examples. The scores therefore provide useful low-resource context, but are not an apples-to-apples leaderboard.

Model	Parameters	Sindhi supervision	Sindhi WikiANN entity F1
Avey-B Sindhi	24.87M	100 target-language training examples; full fine-tuning; 5 seeds	41.59% ± 2.68%
mBERT	≈178M	Zero-shot from English; frozen encoder	8.57%
eBERT	≈186M	mBERT + 10k-token vocabulary extension and Sindhi MLM adaptation; zero-shot NER	21.86%
eBERT + unsupervised alignment	≈186M	eBERT with mined word-pair alignment; zero-shot NER	25.71%

The published comparison uses the same 100-example Sindhi WikiANN resource, but only Avey-B uses its target-language training split. A controlled comparison would require fine-tuning all base encoders on the identical Sindhi split, learning-rate sweep, and seeds; no such unreported scores are presented here.

Other published Sindhi NER references

The following results use different datasets and protocols and must not be numerically ranked against Avey-B's WikiANN result. They are included to make the model card more useful as a map of the small published Sindhi NER landscape.

Model / setup	Evaluation data	Precision	Recall	Entity F1	Token accuracy
mBERT fine-tune	mirfan899/sindhi-ner, reported evaluation set	70.80%	64.43%	67.46%	97.04%
mBERT, direct fine-tuning	Separate gold Sindhi NER corpus	—	—	50%	—
mBERT + machine-labeled pretraining	Separate gold Sindhi NER corpus	—	—	63%	—
XLM-R, direct fine-tuning	Separate gold Sindhi NER corpus	—	—	72%	—
XLM-R + machine-labeled pretraining	Separate gold Sindhi NER corpus	—	—	79%	—

Model-size comparison

Parameter count is directly comparable even when task protocols are not. Avey-B is about 4.5× smaller than the monolingual Sindhi BERT architecture, 7.2× smaller than mBERT, and 11.2× smaller than XLM-R base.

Base encoder	Parameters	Relative to Avey-B	Pretraining scope
Avey-B Sindhi	24.87M	1.0×	Sindhi HPLT 3.0
Sindhi-BERT-base	≈110.7M	4.5×	Sindhi
mBERT	≈178M	7.2×	104-language Wikipedia
XLM-R base	278.04M	11.2×	100-language CommonCrawl

Only Avey-B's latency, throughput, and VRAM were measured in this release. Runtime numbers from other hardware or software stacks are not mixed into the table.

Published MLM context

The independently released Sindhi-BERT-base reports a final perplexity of 31.66 after its S6r training session. Avey-B records 19.01 on its fixed held-out HPLT evaluation. Because the validation text, tokenizer, masking, and evaluation implementation differ, these perplexities are not directly comparable and do not establish a quality ranking.

MLM learning curve

Checkpoint	Loss	Perplexity	Top-1	Top-5
Random initialization	10.3985	32810.46	0.00%	0.01%
Step 1,000	6.6934	807.08	12.13%	21.24%
Step 10,000	3.3303	27.95	42.00%	62.36%
Step 20,000	2.9448	19.01	46.38%	67.39%

Model details

• Architecture: Avey-B masked-language-model encoder
• Parameters: 24,865,804
• Checkpoint size: 94.9 MiB
• Vocabulary: 32,000-token Sindhi WordPiece tokenizer
• Training sequence length: 512 tokens
• Training steps: 20,000
• Effective tokens per optimizer step: 16,384
• Total training-token budget: 327.68M

Training data

The model was trained on the Sindhi snd_Arab portion of HPLT 3.0, using quality bins 10 and 9. The preprocessing pipeline language-filters, PII-redacts, NFC-normalizes, removes exact duplicates, and preserves Sindhi Arabic Yeh (ي).

• Retained training documents: 32,171
• Retained validation documents: 153
• Retained characters: 251,003,680

The complete preprocessing metadata and per-shard counts are included in MANIFEST.json. Machine-readable benchmark measurements and protocols are included in evaluation_results.json.

Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "DanishMahdi/avey-b-snd"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(repo_id, trust_remote_code=True)

Limitations

The training corpus is filtered web text and can contain errors, biases, or harmful content. Evaluation coverage is limited to held-out HPLT masked-token prediction and the very small, automatically annotated Sindhi WikiANN split. Evaluate on native, domain-specific datasets before production use. HPLT licenses its dataset packaging under CC0 but does not own the underlying web documents.