Urdatorn/KainoBERT-sphragis

KainoBERT

KainoBERT is a ModernBERT-base masked language model trained from random initialization for Ancient Greek. It does not inherit the English/code weights of answerdotai/ModernBERT-base; only that model's architecture is reproduced. A fresh 32,768-token byte-level BPE vocabulary was trained on the Ancient Greek corpus.

Corpus and normalization

Training uses Urdatorn/AncientGreek-no-sphragis at revision e021a174ba6526c90736b528e51114734eff976c. Exact normalized lines found in the current Sphragis and Sphragis-Metre benchmarks were removed before pretraining.

All text is lowercased and depunctuated. 80% of records retain polytonic diacritics; a deterministic 20% are transformed with grc_utils.only_bases while preserving word boundaries. This mixture is intended to retain philological orthographic information while making representations less brittle to accent stripping.

Training

• Architecture: ModernBERT base, 22 layers, hidden size 768, 12 heads
• Context used for pretraining: 1,024 tokens
• Objective: masked language modelling, 30% dynamic training masks
• Optimizer: fused AdamW
• Effective batch: 512 sequences
• Initial phase: 3,750 optimizer steps (3.9894 epochs), peak learning rate 3e-4 decaying to 3e-5
• Continuation phase: 14,500 optimizer steps (15.4255 epochs), learning rate 3e-5 with cosine decay
• Effective total: 18,250 optimizer steps (19.4149 epochs)
• Tokens presented to the weights in this release: 9,568,256,000
• Best validation loss: 2.187660
• Best validation perplexity: 8.9143
• Stopping rule: six consecutive validation checks without an improvement larger than 0.002; the 20-continuation-epoch limit was not reached

The released weights are the best checkpoint from the continuation. Full continuation provenance, hyperparameters, and loss history are included in training_metrics.json; the first phase is recorded separately in initial_training_metrics.json. Training code is available in Urdatorn/sphragis_models.

The source is a mixed-license compilation. Consult each source record's license field before redistribution or downstream use.

Limitations

The exclusion procedure establishes zero exact normalized-line overlap with the benchmark versions used to build the corpus. It cannot rule out related editions, paraphrases, or shorter shared phrases. The pristine and repaired source tiers may also contain related versions of the same material.