KainoBERTa
KainoBERTa is a roberta-base masked language model trained from random initialization
for Ancient Greek. It does not inherit the English weights of
FacebookAI/roberta-base; only that model's architecture is reproduced. A fresh
32,768-token byte-level BPE vocabulary was trained on the
Ancient Greek corpus.
Why this model exists
KainoBERTa is one half of an architecture control. Its counterpart
Urdatorn/KainoBERT-sphragis is KainoBERT,
trained on the same corpus, the same packed blocks, the same tokenizer, the same
masking, the same optimizer and schedule, and stopped by the same rule -- six
evaluations without a 0.002 improvement in validation loss. The architecture is
the only thing that differs between them, which is what makes the difference
between them attributable to the architecture rather than to the data.
Corpus and normalization
Training uses
Urdatorn/AncientGreek-no-sphragis
at revision e021a174ba6526c90736b528e51114734eff976c. Exact normalized lines found in the current
Sphragis and Sphragis-Metre benchmarks were removed before pretraining.
All text is lowercased and depunctuated. 80% of
records retain polytonic diacritics; a deterministic 20%
are transformed with grc_utils.only_bases while preserving word boundaries.
This mixture is intended to retain philological orthographic information while
making representations less brittle to accent stripping.
Training
- Architecture: RoBERTa base, 12 layers, hidden size 768, 12 heads, 111,634,688 parameters
- Context used for pretraining: 1,024 tokens
- Objective: masked language modelling, 30% dynamic training masks
- Optimizer: fused AdamW, peak learning rate 3.0e-05
- Effective batch: 512 sequences
- Epochs completed: 34.5745
- Tokens presented: 17,039,360,000
- Best validation loss: 2.871797
Full provenance, hyperparameters, and the loss history are included in
training_metrics.json.
The source is a mixed-license compilation. Consult each source record's
license field before redistribution or downstream use.
Limitations
The exclusion procedure establishes zero exact normalized-line overlap with the benchmark versions used to build the corpus. It cannot rule out related editions, paraphrases, or shorter shared phrases. The pristine and repaired source tiers may also contain related versions of the same material.
- Downloads last month
- -