KFUPM-JRCAI/tnqeet-tokenizer

tnqeet tokenizer

This is the character-level tokenizer used by the tnqeet BiLSTM and Transformer models for Arabic rasm (dot) restoration. It splits text into single characters. The vocabulary has 2,891 entries: Arabic letters, punctuation, other characters seen in the training corpus, and the special tokens [UNK], [PAD], [BOS], [EOS] and [SEP].

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("KFUPM-JRCAI/tnqeet-tokenizer", trust_remote_code=True)
tokenizer.tokenize("لساں الڡٮى")   # ['ل', 'س', 'ا', 'ں', ' ', 'ا', 'ل', 'ڡ', 'ٮ', 'ى']

The tnqeet library (pip install tnqeet) already includes this tokenizer, so you don't need to load it yourself when using the models.

• 🤖 Models: KFUPM-JRCAI/tnqeet-models
• 📄 Paper: Lossy Orthography and Linguistic Inference: Computational Approaches to Arabic Rasm Restoration (EMNLP 2026)
• 💻 Code: https://github.com/MagedSaeed/tnqeet · 📦 PyPI: https://pypi.org/project/tnqeet/
• 🤗 Collection: tnqeet

Citation

@inproceedings{al-shaibani-ahmad-2026-lossy,
    title = "Lossy Orthography and Linguistic Inference: Computational Approaches to {A}rabic Rasm Restoration",
    author = "Al-Shaibani, Maged S.  and
      Ahmad, Irfan",
    booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
    year = "2026",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.emnlp-main.2258/",
    pages = "51351--51371"
}