BDRC/Bo-Yigchung-Detection

mmBERT Yigchung

Token classifier for Yigchung in Classical Tibetan: the small-print interlinear annotation written under the main line. The model reads a flat base string and returns character spans. It does not see page images.

Fine-tune of jhu-clsp/mmBERT-base (ModernBertForTokenClassification, 22 layers, max length 8192) on karma689/yigchung_detection, revision v1.0.

This is a new experiment line (232 books, 3-class BIO).

How to run inference

Spans are inclusive on both ends: text[start:end + 1].

Books are longer than the 8192-token window. Cut the same windows used in training, keep each content token from the first window that covers it, then Viterbi-decode. break_penalty=1 is the value chosen on the validation split.

pip install "transformers>=4.40" torch numpy
git clone https://github.com/Webuddhist-tech/tibetan-yigchung-detection
cd tibetan-yigchung-detection
python src/script/infer_yigchung.py --text-file book.txt

JSON rows look like {"start": 1204, "end": 1388, "text": "..."}.

From Python, after the repo is on PYTHONPATH (run from the clone, or sys.path includes it):

from src.script.infer_yigchung import predict_spans

text = open("book.txt", encoding="utf-8").read()
for start, end in predict_spans(text):
    print(start, end, text[start:end + 1])

predict_spans loads karma689/mmbert-yigchung, uses CUDA when it is available, and defaults to break_penalty=1 and batch size 2. Override with predict_spans(text, device="cpu", break_penalty=1.0).

What the script is doing

Setting	Value
Window length	8192 tokens (2 specials + 8190 content)
Overlap / step	3276 / 4914
Labels	0 O, 1 B-YIGCHUNG, 2 I-YIGCHUNG
Stitch	token is scored by the first window that contains it (later copies were -100 in training)
Decode	Viterbi. O → I is illegal. Leaving a span (B/I → O) costs break_penalty
Selected penalty	1 (validation F1 0.666; test scored once at this value)

A span longer than 8190 content tokens cannot sit inside one window. The longest span in this corpus is 4,823 tokens, so the 40% overlap covers every gold span without rescue windows. Rescue windows were not used at train or test time for this geometry.

Results

Greedy one-to-one span match at IoU ≥ 0.5. Checkpoint is step 2768 (epoch 4.0), the best validation IoU@0.5 F1 during training (in-loop penalty 2.0, F1 0.665). The penalty was then re-chosen on that checkpoint and the test set was scored once.

Split	Precision	Recall	F1	TP	FP	FN
validation, penalty 1	0.665	0.667	0.666	570	287	284
test, penalty 1	0.464	0.402	0.431	351	405	522

Validation F1 by break_penalty on this checkpoint: 0 → 0.659, 1 → 0.666, 2 → 0.665, 4 → 0.665, 8 → 0.655.

The test drop against validation is real. A few test books contribute many false negatives (one book has 107 gold spans and no true positives at penalty 1). Treat 0.431 as the number to cite for this release.

Training

Objective	weighted token cross-entropy, ignore_index=-100
Class weights	O 0.07, B-YIGCHUNG 5.0, I-YIGCHUNG 5.0 (positive weight capped at 5)
Optimizer schedule	AdamW, lr 1e-5, weight decay 0.01, warmup 20% of a 10-epoch reference (not of the 15-epoch budget)
Batch	2 per device, grad accumulation 4, bf16, max grad norm 1.0
Epochs	up to 15, 4 evals per epoch, early stop patience 4 epochs
Selection metric	span IoU@0.5 F1
Seed	42
Attention	Flash Attention 2, SDPA if that kernel is unavailable

Train / validation / test windows: 5,546 / 696 / 722. About 78% of train windows contain no Yigchung token. Overlapped tokens are labelled in exactly one window.

Data

232 reviewed Classical Tibetan books, 8,495 Yigchung spans, 51,535,768 characters. Book-level 80/10/10 split (184 / 24 / 24). A book never appears in two splits. Splits were stratified by catalog family (P / I) and span-count bin, then swapped inside a stratum until span density matched (coverage ratio 1.00× across splits).

Split	Books	Spans	Text chars	Chars in spans
train	184	6,768	41,027,735	1.788%
validation	24	854	5,178,401	1.789%
test	24	873	5,329,632	1.789%

Gold config on the Hub is the book text plus spans. The windowed config windowed_w8192_s4914 is derived from tokenizer jhu-clsp/mmBERT-base and this geometry. Do not mix it with windows from another tokenizer or stride.

Label scheme

id	label
0	O
1	B-YIGCHUNG
2	I-YIGCHUNG

The published windows are already these three ids. Do not remap them from a 15-class head. Label -100 is the loss mask for specials and for tokens already owned by an earlier window; it is not a class.

License and attribution

Base model license follows jhu-clsp/mmBERT-base (Apache-2.0). Source texts were digitized and made available by the Buddhist Digital Resource Center (BDRC). Annotations were prepared through OpenPecha with support from the Tsadra Foundation.