normaere-ByT5

Transformer-based seq2seq normalizer for Middle High German (MHG) text, fine-tuned from google/byt5-base.

Model Details

• Base Model: google/byt5-base (580M parameters)
• Architecture: T5ForConditionalGeneration (encoder-decoder)
• Precision: FP16
• Max Sequence Length: 1500 tokens
• Vocabulary Size: 384 (byte-level tokenizer — no subword vocabulary)

Byte-Level Tokenization

Unlike the mT5 variant (JonasHermann/normaere-model), this model uses ByT5's byte-level tokenizer (~1 token per character vs. ~0.35 tokens per character for SentencePiece models). For long inputs, use a stride window of ~1300 tokens with a step of ~1000 tokens to match the training-sample length distribution (training samples were up to 1200 characters ≈ 1350 tokens).

Intended Uses

Normalizes Middle High German texts (ca. 1050–1500) according to the standards of the Referenzkorpus Mittelhochdeutsch (ReM):

• Solves common abbreviations
• Splits off and expands common pro- and enclitics
• Applies editorial post-processing

Training Data

Fine-tuned on the Referenzkorpus Mittelhochdeutsch (ReM), Version 2.1.

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("JonasHermann/normaere-ByT5")
model = AutoModelForSeq2SeqLM.from_pretrained("JonasHermann/normaere-ByT5")

input_text = "die stete wârheit"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs)
result = tokenizer.decode(outputs[0], skip_special_tokens=True)

References

• ByT5: Xue et al., 2022
• Training corpus: Roussel, Adam; Klein, Thomas; Dipper, Stefanie; Wegera, Klaus-Peter; Wich-Reif, Claudia (2024). Referenzkorpus Mittelhochdeutsch (1050–1350), Version 2.1, https://www.linguistics.ruhr-uni-bochum.de/rem/. ISLRN 937-948-254-174-0.