chagatai-project/chagatai-sentence-segmentation

Chagatai Sentence Boundary Detection (SBD) [Stanza Models]

Neural Sentence Boundary Detection (SBD) models for Chagatai (Chaghatay / جغتای), a classical Turkic literary language written in Perso-Arabic script, based on the Stanford Stanza tokenization and Character Language Model (CharLM) architecture.

Classical Chagatai texts (such as Babur's Baburnama and Navoi's poetry) were written without modern western punctuation marks. Sentence boundary detection is the foundational first step for machine translation, parsing, tokenization, and linguistic corpus analysis.

All models are compiled into single, self-contained ONNX files with in-graph ensembling and dynamic quantization, requiring zero PyTorch or Stanza dependencies at inference time.

---

📊 Benchmark & Evaluation Results

Evaluated on the out-of-sample Chagatai test set (295 paragraphs, 71,673 character tokens, 868 true sentence boundaries):

Model	Format	Size	F1	Precision	Recall	Exact Match	Key Benefit
Tri-Hybrid [Stanza]	FP32	53.3 MB	74.80%	79.95%	70.28%	27.46% (81/295)	Best F1 & Precision (lowest false positives)
Tri-Hybrid [Stanza]	INT8	13.5 MB	74.71%	79.74%	70.28%	27.12% (80/295)	~4x compressed, loss of only 0.09% F1
BiCharLM [Stanza]	FP32	82.8 MB	74.59%	79.17%	70.51%	27.80% (82/295)	Highest Exact Match record
BiCharLM [Stanza]	INT8	26.0 MB	74.42%	78.94%	70.39%	27.80% (82/295)	3.2x compressed, 100% EM preserved
Standalone [Stanza]	FP32	29.2 MB	71.94%	71.09%	72.81%	22.37% (66/295)	Fastest single model (no ensemble)

• Exact Match (EM): percentage of full paragraphs (2 to 5 sentences each) segmented with zero errors.
• INT8 Quantization: dynamic quantization shrinks the Tri-Hybrid model by 74.7% (down to 13.5 MB) with virtually zero loss in accuracy.

---

📦 Model Files & Recommended Thresholds

File	Format	Size	Threshold	Recommended For
chagatai_sbd_tri_hybrid_stanza.onnx	FP32	53.3 MB	tau = 0.42	Highest F1 score and precision
chagatai_sbd_tri_hybrid_stanza_int8.onnx	INT8	13.5 MB	tau = 0.42	Ultra-lightweight production deployment
chagatai_sbd_bicharlm_em_stanza.onnx	FP32	82.8 MB	tau = 0.43	Highest paragraph-level exact match
chagatai_sbd_bicharlm_em_stanza_int8.onnx	INT8	26.0 MB	tau = 0.43	Compact exact-match ensemble
chagatai_sbd_fwd_charlm_stanza.onnx	FP32	29.2 MB	tau = 0.24	High-throughput single model
vocab.json	JSON	2.4 KB	—	Character vocabulary mapping
inference.py	Python	5.2 KB	—	Standalone inference script

---

🚀 Quickstart (Zero-Dependency Python Inference)

No PyTorch, Transformers, or Stanza required. All you need is onnxruntime and numpy:

pip install onnxruntime numpy huggingface_hub

Segmentation Example

from huggingface_hub import hf_hub_download
import importlib.util

REPO = "chagatai-project/chagatai-sentence-segmentation"

# 1. Download model, vocabulary, and inference script (using lightweight 13.5 MB INT8 model)
model_path = hf_hub_download(repo_id=REPO, filename="chagatai_sbd_tri_hybrid_stanza_int8.onnx")
vocab_path = hf_hub_download(repo_id=REPO, filename="vocab.json")
code_path = hf_hub_download(repo_id=REPO, filename="inference.py")

# 2. Load inference engine
spec = importlib.util.spec_from_file_location("inference", code_path)
inf = importlib.util.module_from_spec(spec)
spec.loader.exec_module(inf)

# 3. Initialize detector
sbd = inf.ChagataiSBD(model_path=model_path, vocab_path=vocab_path)

# 4. Segment historical Chagatai text (Baburnama)
text = "سلطان ابوسعید میرزا شهادت تاپقاندین سونگ سمرقند ولایتینی میرزا احمد آلدی و هرات ولایتینی سلطان حسین میرزا ضبط قیلدی و اندیجان ولایتینی عمرشیخ میرزا توتتی"

sentences = sbd.segment(text)
for idx, sent in enumerate(sentences, 1):
    print(f"[{idx}] {sent}")

Output:

[1] سلطان ابوسعید میرزا شهادت تاپقاندین سونگ سمرقند ولایتینی میرزا احمد آلدی
[2] و هرات ولایتینی سلطان حسین میرزا ضبط قیلدی
[3] و اندیجان ولایتینی عمرشیخ میرزا توتتی

---

🛠️ Architecture Details [Stanza Framework]

Each ONNX model compiles the complete Stanza tokenization and segmentation pipeline into a unified computational graph:

1. Stanza Character Language Models (CharLM):
   • Forward & Backward 512-dimensional LSTMs trained unsupervised on historical Chagatai text.
   • Captures sub-word morphology and archaic Turkic suffixes without dictionary lookups.
2. Hierarchical BiLSTM Backbone:
   • 3-layer Bidirectional LSTM backbone with character embeddings and positional features (space_before, is_capitalized, is_numeric, start_of_para, end_of_para).
   • Second hierarchical BiLSTM conditioning sentence boundaries on tokenization logits.
3. Cross-Lingual Turkic Transfer Learning:
   • Pre-trained on a balanced multi-source dataset (Chagatai + South Uzbek + Uyghur) preventing overfitting on rare characters.
4. In-Graph Ensemble Fusion:
   • Fuses models with complementary inductive biases (Domain purity + Weighted loss + BiCharLM).
   • In-graph soft-voting probability averaging and thresholding executed natively within ONNX Runtime.