ContinuaRoformer-slav

A model trained from scratch on Old East Slavic and Church Slavonic texts that restores damaged text, attributes its dialect and date, and puts word boundaries back. It reads scriptio continua — text without spaces.

Architecture

• Input: characters without spaces; punctuation folded into one divider ·. Each position also gets the id of the character 3-gram starting there (3-grams touching a gap or damage marker are blanked, so nothing leaks). Char (768) and 3-gram (192) embeddings are concatenated and projected to 768.
• Encoder: 10 post-LN transformer layers, hidden 768, 8 heads, rotary position embeddings, 512-character context.
• Heads: character restoration (tied to the char embeddings), # gap expansion (expand / stop), word boundary after each character, date (20 bins of 50 years, 800-1799), dialect (OES / CS / NW / SW). Date and dialect read a learned attention pool over all text positions.

Usage

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("BeRestoral/ContinuaRoformer-slav")
sys.path.insert(0, path)
from inference import ContinuaRestorer

r = ContinuaRestorer(path, f"{path}/char_vocab.json", f"{path}/ngram_vocab.json")
# `-` one missing character, `#` a gap of unknown length, `?` damage not to restore
print(r.restore_text("поклоноѿд-выдакъмат#ию"))                 # with predicted spaces
print(r.restore_text("поклоноѿд-выдакъмат#ию", segment=False))  # continua
print(r.attribute("поклоноѿдавыдакъматѳию"))                    # dialect and date distributions

Requires torch and transformers. The model was trained on normalized text, so input should already use the model's alphabet (char_vocab.json). Any other character is read as damage ([UNK]).

Evaluation

• Real lacunae (Test B): restoration Hit@1 = 28.5%, Hit@5 = 64.4% (3616 editorially reconstructed characters in birchbark letters and inscriptions, known length, spaces not counted).
  • Word boundaries: accuracy = 91.9%, F1 = 82.5% (24.0% of characters are followed by a space, so never predicting a boundary scores 76.0%); inside lacunae F1 = 52.2%.
  • Dialect: accuracy = 92.0%, macro-F1 = 61.6% (NB: over four classes, however only two of them, NW and CS, are present).
  • Date: exact 50-year bin = 35.8%, MAE = 65.9 years.
• Synthetic damage (Test A): restoration Hit@1 = 40.9%, Hit@5 = 74.2%; word boundaries F1 = 84.9%; dialect accuracy = 90.2%; date exact bin = 47.2%, MAE = 74.7 years.
• Unknown-length gaps, real lacunae (Test B) (1164 gaps, each hidden behind one # and decoded iteratively): exact match by length 1: 28.1%, 2: 12.7%, 3: 0.8%; 16.1% keep expanding until the step limit.
• Unknown-length gaps, synthetic 1-10 characters (Test A) (228 gaps, each hidden behind one # and decoded iteratively): exact match by length 1: 46.2%, 2: 21.7%, 3: 14.3%; 8.3% keep expanding until the step limit.

Limitations

• Gaps of unknown length are usable only for 1-2 characters; from 3 on the predicted length is almost never right, and the gap head often over-expands (it is trained on the first expand/stop decision only).
• Dialect and date attribution partly rely on genre and source, and on the density of [UNK] damage markers in birchbark letters.