DualEmbRoformer-slav

A masked language model trained from scratch on Old East Slavic and Old Church Slavonic texts. Combines RoPE-BERT torso and unknown-length lacuna restoration with DualEmbLM's word-level embedding channel.

Architecture

• Character-level tokenization (1 character = 1 token) for precise lacuna restoration
• Word-level context embeddings via a 50k-word vocabulary (DualEmbLM's dual-embedding scheme), concatenated with the char embedding and projected back to hidden_size
• Rotary Position Embeddings (RoPE)
• 4 task heads: character restoration (local, tied to char_embeddings), unknown-length gap expansion (local, binary), date-bin classification (global, pools from [SOS]), region/dialect classification (global, pools from [SOS])
• hidden_size=768, 10 layers, 8 attention heads

Training

Trained on Old Russian/Church Slavonic sources: NKRYA, UD Old East Slavic (RNC + Ruthenian), epigraphica, birchbark manuscripts, TOROT, Pushkin House texts, Sofia chronicle, byliny, and the Ostrog Bible. byliny/Ostrog Bible carry no date/dialect label (withheld from those two losses to avoid skewing the classifiers toward a single year/dialect) -- text-only additions for the restoration task.

Masking: per-document variable char-mask rate + non-compressing spans + one compressed unknown-length gap + simulated torn edges, plus a "balanced" per-bin reweighting of the date KL-divergence loss to counter bin imbalance (macro-F1 on rare bins was being starved of gradient under a flat per-example average).

Usage

import json, torch
from transformers import AutoConfig, AutoModel

repo = "BeRestoral/DualEmbRoformer-slav"
config = AutoConfig.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True)
model.eval()

char_vocab = json.load(open("char_vocab.json"))
word_vocab = json.load(open("word_vocab.json"))

from vocab_categories import tokenize_text
from dualemb_collator import load_word_vocab, DualEmbRoformerCollator

word_vocab_map = load_word_vocab("word_vocab.json")
collator = DualEmbRoformerCollator(char_vocab=char_vocab, word_vocab=word_vocab_map, mode="valid")

text = "от борис[-] къ ма[#]"  # [-] = single-char mask, [#] = unknown-length gap
input_ids = torch.tensor([tokenize_text(text, char_vocab)])

Tasks

• Real lacunae restoration (Test B) — Hit@1 = 0.3559, Hit@5 = 0.7011 (single-pass per-position accuracy, 3,911 real editorial-bracket positions; every gap here has a known length — each missing character is revealed as an individual [-] mark).

  • Date classification: MAE = 63.44 years, Accuracy = 0.3584, Macro F1 = 0.2096.
  • Region classification: Accuracy = 0.9177, Macro F1 = 0.6114.

• Synthetic masking (Test A) — dynamic per-example masking, same single-pass per-position accuracy as in Test B: Hit@1 = 0.4158, Hit@5 = 0.7033 (1,318 masked positions; masking also injects at most one compressed unknown-length [#] gap per document, which the Unk head must expand/resolve).

  • Unk head ([#] expand/stop decision, single-step accuracy): Accuracy = 0.8071, Macro F1 = 0.7518.
  • Date classification: MAE = 56.97 years, Accuracy = 0.5139, Macro F1 = 0.3806.
  • Region classification: Accuracy = 0.9529, Macro F1 = 0.9396.

Contact

Maxim Eremeev, maxim.a.eremeev@gmail.com