HabsburgLLM

A research project to train an LLM from scratch using only historical data. The goal is knowledge creation and research, not producing a production-quality language model. The focus is on learning what happens when a model is trained exclusively on historical texts. Not instruction-tuned yet. Instruction fine-tuning is in the works. Coming Soon™.

Code and training setup: https://github.com/m2key1/HabsburgLLM

Models

Model	Where	Params	Language	Data	Weights
olmo-habsburg	repo root, main	3.97B	German	Austrian newspapers, 1800 to June 1914	fp32
olmo-habsburg steps	branches habsburg-step-*	3.97B	German	same	bf16
olmo-london	olmo-london/final	547M	English	London texts, 1800 to 1875	fp32

All are OLMo 3 architecture (Olmo3ForCausalLM), each with its own byte-level BPE tokenizer.

olmo-habsburg - 32 layers, hidden size 3072, GQA (24 query / 8 KV heads), 4096-token context, 48k vocabulary. 5 epochs (23,285 steps, about 114B tokens seen) over 22.9B tokens of German-language Austrian newspapers from ANNO (Austrian National Library), 1800 to June 1914, re-OCRed for clean text. Trained on 8 NVIDIA H200 GPUs for 13.9 days on the DHinfra.at cluster in Graz. Final validation loss 1.910 (perplexity 6.75) on a held-out split of the same corpus.

olmo-london - 24 layers, hidden size 1024, 2048-token context, 32k vocabulary, trained on postgrammar/london-llm-1800.

Revisions

The final olmo-habsburg model is at the root of main (step 23,285). Intermediate checkpoints of the same run sit at the root of these branches:

habsburg-step-1000, habsburg-step-5000, habsburg-step-10000, habsburg-step-15000, habsburg-step-20000, habsburg-step-23000

Each includes trainer_state.json with the loss history up to that step.

Usage

Tested with transformers 5.9.0.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "M2key1/HabsburgLLM"
device = "cuda" if torch.cuda.is_available() else "cpu"

def generate(prompt, subfolder=""):
    tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=subfolder)
    model = AutoModelForCausalLM.from_pretrained(repo, subfolder=subfolder, dtype=torch.bfloat16).to(device)
    inputs = tokenizer(prompt, return_tensors="pt").to(device)
    output = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        no_repeat_ngram_size=6,
    )
    return tokenizer.decode(output[0], skip_special_tokens=True)

print(generate("Wien, 27. Juni."))                                    # olmo-habsburg
print(generate("London, June 1851.", subfolder="olmo-london/final"))  # olmo-london

Intermediate checkpoint: add revision="habsburg-step-5000" to both from_pretrained calls.

Limitations

• Base model: it continues text in period style rather than answering questions. Asked about events, it reproduces the genre where such text would appear, mixing real and invented detail in a convincing register.
• It almost never emits an end-of-sequence token, so generation runs to max_new_tokens.
• Greedy and beam decoding fall into loops; use sampling with no_repeat_ngram_size=6.
• It reflects the views, language and biases of its sources.

Citation

@software{fleischhacker_habsburgllm,
  author = {Fleischhacker, David},
  title  = {HabsburgLLM},
  year   = {2026},
  url    = {https://github.com/m2key1/HabsburgLLM}
}