ArieLLL123/judaic-semantic-tokenizer-v0.1

Judaic Semantic Tokenizer v0.1-candidate

A compact Unigram + Metaspace tokenizer for semantic retrieval across rabbinic Hebrew, Aramaic, modern Hebrew, and limited Judeo-Arabic. This is a complete trained tokenizer candidate, not a sample. It is not an embedding model and this repository intentionally contains no encoder weights.

Training summary

• base vocabulary: 24,000 tokens, including all seven special/role tokens;
• training sample: 353,889 normalized texts (322,942 local-domain texts + 30,947 modern-Hebrew HeQ training texts);
• held-out evaluation: 13,428 texts;
• held-out unknown-token fraction: 9.986568e-7 (0.0000999%);
• held-out mean token/word ratio: 1.543809;
• held-out p50/p95 token/word ratio: 1.46087 / 2.25;
• modern-Hebrew query ratio improved from 1.9915 to 1.7835 after adding HeQ;
• modern-Hebrew passage ratio improved from 2.0492 to 1.8504.

It is labeled v0.1-candidate, rather than final, because it has not yet been compared against 16k/32k vocabularies on end-to-end retrieval metrics and was sampled from the local 7.13M-line collection instead of trained on every line.

Mandatory normalization

Use preprocessing.normalize_text before tokenization. The preprocessing keeps visible HTML text but removes HTML tags; applies NFKC; strips Hebrew diacritics, Latin letters, bidi/invisible controls; canonicalizes common punctuation variants; and collapses whitespace. The tokenizer JSON alone does not reproduce all of these domain rules.

from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer
import importlib.util

repo_id = "ArieLLL123/judaic-semantic-tokenizer-v0.1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
preprocessing_path = hf_hub_download(repo_id, "preprocessing.py")
spec = importlib.util.spec_from_file_location("preprocessing", preprocessing_path)
preprocessing = importlib.util.module_from_spec(spec)
spec.loader.exec_module(preprocessing)

text = preprocessing.normalize_text("<b>בְּרֵאשִׁית</b> bara")
encoded = tokenizer("[שאילתה] " + text, truncation=True, max_length=256)

Special tokens: [PAD], [UNK], [CLS], [SEP], [MASK], [שאילתה], [קטע].

Intended use and limitations

Use as the tokenizer for a compact bi-encoder trained with the associated aligned-pair dataset. It is designed for unvocalized retrieval. Niqqud-sensitive tasks, Latin-heavy bibliographic queries, and unrelated languages are out of scope. Tokenization efficiency is only a proxy; the final choice must be made using retrieval recall/MRR/nDCG and latency.

Data and license

See LICENSE.md and training_metadata.json. The tokenizer was trained on a mixed-rights private corpus and should remain private pending the local-text rights audit. The HeQ-derived modern-Hebrew component is CC BY 4.0.