KainoBERT

KainoBERT is a ModernBERT-base masked language model for Ancient Greek, trained from random initialization with a fresh 32,768-token byte-level BPE vocabulary (22 layers, hidden size 768, 12 heads, 136M parameters, 1,024-token context). It does not inherit the English weights of answerdotai/ModernBERT-base; only the architecture is reproduced.

This is the un-audited, most complete version of KainoBERT, for use in the wild. It has read the whole of Ericu950/AncientGreek, including every work in the Sphragis and Sphragis-Metre authorship benchmarks. It must not be evaluated on those benchmarks. For controlled, contamination-free experiments use Urdatorn/KainoBERT-sphragis, which is this model's parent and has never seen a benchmark work, passage or line.

How it was trained

Three phases, all with 30 % dynamic masking, fused AdamW, bf16, an effective batch of 512 sequences of 1,024 tokens, on two GH200 GPUs.

1. Pretraining on Urdatorn/AncientGreek-no-sphragis (second derivative, revision v2-sphragis-1e6d8b58d956e84aec7c1c778bef036bd0286fa9): AncientGreek with every benchmark work, every other edition or commentary of one, every record carrying a benchmark window and every benchmark line removed. 323,225 training blocks (331M tokens), peak learning rate 3e-4, cosine schedule, four epochs.
2. Continuation on the same corpus from the terminal learning rate of phase one (3e-5, fresh optimizer), stopped after six evaluations without a 0.002 improvement in validation loss (epoch 18.2, loss 2.380). The result is Urdatorn/KainoBERT-sphragis.
3. Continuation on the removed material (this model): everything phase one's corpus had excluded, reconstructed by diffing AncientGreek (revision 6ac90787c669a7e9218d6d4675a029fa3f10ed99) against the second derivative: 313,840 records dropped whole (750M characters, both quality tiers) plus the 4,092 removed lines of records that were kept. Packed with the same tokenizer and surface rules (lowercased, depunctuated, 80 % polytonic, 20 % bare letters) into 161,745 blocks (166M tokens), 0.2 % held out for validation. Same recipe as phase two: learning rate 3e-5, no warmup, cosine to 10 %, early-stopping rule 6 × 0.002, ceiling 30 epochs. Run in legs of 30 epochs, each restarting the cosine schedule from 3e-5 with a fresh optimizer, as phase two did after phase one. Validation loss on the removed material: 2.614 for the phase-two model (unseen text); 1.977 after leg 1; 1.806 after leg 2; 1.706 after leg 3 (this revision). A leg is added while a fresh schedule still buys more than the stopping rule's margin; the Hub revision history keeps every leg.

Full hyperparameters, the packed-corpus manifest and the loss history are in training_metrics.json. The pipeline is in Urdatorn/sphragis_models (scripts/build_removed_corpus.py, scripts/prepare_removed_corpus.py, slurm/continue_kainobert_v3.slurm).

Input surface

Text should be lowercased and depunctuated, polytonic marks kept, as in sphragis_models.kainobert_corpus.polytonic_surface; the model also reads bare-letter text. The tokenizer adds [CLS] … [SEP].

Licence

The corpus is a mixed-licence compilation. Consult each source record's license field and the upstream dataset cards before reuse.

Created by Albin Thörn Cleland, Lund University (ORCID 0009-0003-3731-4038).