Nanite-Labs/nanites-medieval-e2b-chat

nanites-medieval-gemma4-e2b

A Gemma-4-E2B model fine-tuned to speak as a literate person of late-medieval England — a 15th-century letter-writer — drawn from real pre-1500 letters and papers. You address it as a modern visitor; it answers in a modernized but period register.

What it does

Ask it like a visitor asking a 1400s gentleman about the harvest, the wool market, a lawsuit, the war in France, or a daughter's marriage. It answers in 1-3 sentences with period vocabulary — "woe worth the weather", "thou", "hath", "groat", "staple", "good cousin" — never verbatim Middle English.

visitor> How fares your household this season?
manuscript> I thank you, the household is whole and my wife hath borne
      the long winter well, though the roofs of the great hall leak
      and I must send the carpenter before harvest.

visitor> What news of the war in France?
manuscript> Letters say the garrison at Calais is lean and the king
      demands a subsidy to pay it. Money is the sinew of war, and the
      Exchequer is dry.

visitor> My neighbour hedges in the common.
manuscript> Hue and cry upon him at the manor court, and bring the
      old men that remember the true bounds. Such inclosure without
      licence be a felony against the commons.

Multi-turn works — the correspondent sees your prior exchanges in context (Gemma chat template).

Training

• Base model: unsloth/gemma-4-E2B-it (multimodal Gemma-4; only language layers fine-tuned). We train on a text-only prequantized local repack (vision tower dropped, all 245 transformer linears packed NF4) rather than the mixed checkpoint.
• Method: QLoRA (4-bit base, LoRA r=16 alpha 32 on q,k,v,o,gate,up,down), use_cache=True for inference (required for E2B/E4B KV-shared layers)
• Framework: transformers 5.5.0 + peft (plain LoraConfig), custom trainer loop (unsloth's get_peft_model is unsupported for this arch)
• Hardware: NVIDIA RTX 4070 Laptop (8GB VRAM). Gemma-4's per-layer-input embeddings (embed_tokens_per_layer, 262144 x 8960) put the base at ~6.6 GB GPU; they are kept frozen on CPU with a patched embedding forward (lm_head untied onto GPU) so real-length sequences fit. Validated peaks: ~6.5 GB at training, ~6.5 GB inference.
• Data: 50 hand-authored seed Q&A + ~200 two-teacher distilled pairs (laguna-xs-2.1 + ornith, both local ollama) grounded in a parsed corpus of pre-1500 letters and verse + 3 multi-turn conversations (7 progressive samples). 232 train / 25 val samples.
• Epochs: 2, batch 1, grad accum 4, lr 2e-4 cosine, max seq 512 (train loss 14.2 -> ~5.1, val loss 3.42)
• Output: adapter only (inference loads base + adapter)

Prompt format

Gemma-4 chat template (non-thinking). Use the chat REPL in the source repo (python -m experts.medieval.chat) which applies the template and trims generation. Load the base as a 4-bit bf16 model with AutoModelForCausalLM.from_pretrained(base, device_map={"":0}, dtype=torch.bfloat16), apply the freezing-embedding-offload patch from experts/medieval/train.py, wrap with PeftModel.from_pretrained(..., "Nanite-Labs/nanites-medieval-gemma4-e2b"), and generate with use_cache=True.

Intended use

• In-domain: casual conversation with a 15th-century English letter-writer — estates, lawsuits, wool and trade, war and kings, household, marriage, sickness, weather, proverbs. Multi-turn works.
• Out-of-domain: no fact-check layer. Names, dates, places, legal details are creative writing in a historical register, not history.
• Not for: settling real historical questions or citation.

Limitations

• Thin character sketch: period vocabulary is grounded in the corpus, but world knowledge is shallow and anachronisms persist occasionally ("hold your horses", "audit", "business", "team"). Regenerate to dodge them.
• Short generations only (1-3 sentences); long rambles may loop.
• The base model is multimodal but only the text tower is bundled / fine-tuned; no vision.
• No guardrails. It will give period-style advice with confidence; do not act on it.

Provenance

Training corpus (data/parsed/corpus.jsonl, 4,172 entries / ~4.3 MB):

• CEECS (Corpus of Early English Correspondence Sampler, Oxford University via the Oxford Text Archive): 242 pre-1500 letters kept. Research license — not redistributed.
• Paston Letters A.D. 1422-1509, Vol. 1 of 6 (ed. James Gairdner), Gutenberg ebook 43348.
• Chaucer — Canterbury Tales & minor poems (D. Laing Purves edition), Gutenberg ebook 2383, rich in dialogue.
• Gower — Confessio Amantis (ebook 266); Langland — Piers Plowman (ebook 436); Gawain — Sir Gawayne and the Green Knight (ebook 14568); Lydgate — The Disguising at Hertford (2878) and The Temple of Glass (29552); Hoccleve — Works (EETS 1897), from an Internet Archive scan of the 1897 edition.
• All Gutenberg/IA texts public domain. Corpus is reproducible via experts.medieval.acquire + experts.medieval.parse.

The ~200 distilled Q&A pairs were produced by two local teachers — laguna-xs-2.1 and ornith (neither devstral nor nemotron, the earlier experts' teachers) — prompted with parsed letter/verse excerpts. The 50 seed pairs and multi-turn conversations were written by hand as a style anchor.

License

Code & model: Apache 2.0. Training data: not redistributed; users can re-scrape via experts.medieval.acquire.