Ericu950/CharDiff-grc-doc_clean

CharDiff-grc (documentary-clean)

A 405M-parameter character-level masked-diffusion Transformer for Ancient Greek, pretrained with zero exposure to any documentary text — a leak-proof base for epigraphic and papyrological fine-tuning, and the strongest of eleven released CharDiff-grc backbones across every downstream task evaluated (restoration, morphosyntactic tagging/parsing, macronization/scansion).

This is one of eleven backbones released together: ten rotated, work-level-decontaminated literary folds (CharDiff-grc-fold-0 … CharDiff-grc-fold-9) plus this documentary-clean model. For any passage in the open training corpus, at least one of the eleven has provably never seen it during pretraining.

Architecture

The input is not word- or subword-level tokens but four independently-maskable character-level planes (a fifth, capitalization, is output-only in this base model):

plane	meaning	vocab
input_ids (chars)	letter id, 24-letter minimal alphabet (final/lunate sigma merged)	27 (24 letters + mask + blank + pad)
boundary	0=word-internal / 1=word-final / 2=sentence-final	4 (+ unknown)
dia	diacritic state: accent × breathing × iota-subscript × diaeresis	49 (48 states + unknown)
punct	punctuation category following this letter	7 (6 classes + unknown)

Any subset of these planes can be masked (set to its "unknown" id) at inference time, and — critically — any combination of them can be masked jointly in one forward pass: a gap of missing letters, unaccented surrounding text, and unmarked word divisions can all be reconstructed together, which is the realistic case for damaged, unaccented, unspaced primary sources (inscriptions, papyri, majuscule manuscripts). See Usage below for the joint-reconstruction and uncertain-lacuna-width examples this backbone actually supports. For optimal performance, the model should be finetuned on a specific corpus.

405M params, d_model=1024, depth 32, alternating local (256-character window) and global attention (3:1 ratio), QK-norm, RoPE. Published with attn_impl="sdpa" (portable CPU/GPU path); the original training used a compiled FlexAttention block-sparse path for long packed sequences, not needed for standalone inference on single passages.

Usage

import torch
from transformers import AutoModel
from huggingface_hub import hf_hub_download

hf_hub_download(
    repo_id="Ericu950/CharDiff-grc-doc_clean",
    filename="processing_char_bert.py",
    local_dir=".",
)

model = AutoModel.from_pretrained("Ericu950/CharDiff-grc-doc_clean", trust_remote_code=True)
model.eval()

# the processor wraps text <-> the four planes above (see full usage example in the repo)
from processing_char_bert import CharBertProcessor
processor = CharBertProcessor()

# Joint restoration: a lacuna of KNOWN width ('-' runs) inside text that is ALSO
# fully bare scriptio continua (no spaces, no accents) -- letters (inside the gap),
# accents, and word boundaries are all masked and predicted together, everywhere,
# not just inside the gap. This is the realistic case for damaged, unaccented,
# unspaced primary sources.
damaged = "εναρχηηνολογ--καιολογοςηνπροστονθεον"
batch = processor(damaged, mask_planes=["dia", "boundary"], has_boundaries=False)
with torch.no_grad():
    out = model(**{k: v for k, v in batch.items() if k != "_cap"})
print(processor.decode_restoration(out, batch))
# -> ἐν ἀρχῇ ἦν ὁ λόγος, καὶ ὁ λόγος ἦν πρὸς τὸν θεόν.

# A lacuna of UNCERTAIN width -- the realistic editorial case, since editors estimate
# a lacuna's length and rarely know it exactly. Write "[N±M]" for a best guess N and
# a plausible range N-M..N+M; every candidate width is scored by the model's own
# confidence in its letter predictions inside the gap, so this recovers both the
# width AND the text -- while still jointly restoring accents/boundaries everywhere.
text = "εναρχηηνο[5±3]καιολογοςηνπροστονθεον"
best_text, best_width, candidates = processor.restore_elastic(model, text, mask_dia_boundary=True)
print(best_text, " (width:", best_width, ")")
for w, filled, logp in candidates:
    print(f"  width {w}: {filled}  (logp={logp:.3f})")
# -> width 5: ἐν ἀρχῇ ἦν ὁ λόγος, καὶ ὁ λόγος ἦν πρὸς τὸν θεόν.  (logp=-0.0001,
#    next-best candidate width 6 at logp=-0.2389 -- correct width by a huge margin)

# Single-plane variants (decode_diacritics / decode_boundaries) are also available
# for narrower use cases -- see the repo's processing_char_bert.py for details. Note
# that testing decode_boundaries on text that ALREADY has real accents isn't a
# meaningful standalone test: Greek marks exactly one accent per word, so an
# already-accented input hands the boundary head most of the answer for free.

Downstream results (this checkpoint)

task	result
Restoration (inscriptions, strict protocol, CER)	0.178
Restoration (papyri, combined w/ inscriptions, CER)	0.098
Morphosyntax (fine-tuned joint tagger+parser, composite score, mean±std, 5-fold×2-seed)	0.9196

Fine-tuned downstream models built on this backbone are released separately: Ericu950/CharDiff-grc-restoration, Ericu950/CharDiff-grc-tagger-parser, Ericu950/CharDiff-grc-meter.

Training data

Pretrained on Ericu950/AncientGreek and Ericu950/SyntheticAncientGreek-CorpusCorporum (both public), minus every inscription/papyrus record and every literary passage echoing one. See the paper (citation below) for full corpus and decontamination methodology.

Citation

See the CharDiff-grc repository and the associated paper (citation to be added on publication).