Evicka/Hanse2-100M-Base

Hanse2-100M Base

German–English base language model · 99.1M parameters · 20B pretraining tokens · up to 8K context

Hanse2-100M-Base is a base model, not a chat assistant. It has not been instruction-tuned, preference-tuned, or safety-aligned. Use it as a text-completion model or as a starting point for continued pretraining and post-training.

Hanse2-100M-Base is a 99,144,320-parameter decoder-only causal language model trained from scratch on 19,999,752,192 German and English tokens. It uses a compact Llama-style architecture, a custom 32K byte-level BPE tokenizer, grouped-query attention, and a three-stage pretraining curriculum that progressively extends context length from 2K to 8K.

The model was trained on a single AMD Radeon RX 9070 XT 16 GB, and the repository includes the tokenizer-training, pretraining, and inference scripts used for the project.

Highlights

• 99.1M parameters, trained from scratch
• 19.999B pretraining tokens (~201.7 tokens per parameter)
• German + English, approximately 56% / 44% by planned token share
• 2K → 4K → 8K progressive context curriculum
• 32K custom byte-level BPE tokenizer
• Single-GPU training on an AMD Radeon RX 9070 XT 16 GB
• Reproducible project code for tokenizer training, pretraining, and inference
• Apache-2.0 release

Model overview

Model type	Decoder-only causal language model
Architecture	Llama-style Transformer
Parameters	99,144,320
Languages	German, English
Vocabulary	32,000
Training tokens	19,999,752,192
Tokens / parameter	~201.7
Configured context	8,192 tokens
Training precision	bfloat16
Position encoding	RoPE, θ = 10,000
Embeddings	Tied input/output embeddings
Instruction tuned	No
Safety aligned	No

Architecture

Hyperparameter	Value
Hidden size	640
Layers	16
Attention heads	10
KV heads	2
Head dimension	64
Intermediate size	2,048
Vocabulary size	32,000
RoPE θ	10,000
Tied embeddings	Yes

Pretraining

Pretraining used three consecutive phases while keeping the effective optimizer batch at approximately 131,072 tokens per optimizer step.

Phase	Purpose	Tokens	Context	Micro batch	Grad. accum.	Learning rate
1	Broad bilingual pretraining	~15B	2,048	2	32	1% warmup → constant 6e-4
2	Quality-focused annealing	~4B	4,096	1	32	1% warmup, 3e-4 → 3e-5, 1-sqrt decay
3	Cooldown + context extension	~1B	8,192	1	16	cosine 3e-5 → 1e-5

Data mixture

Approximate token share over the full curriculum:

Source	Share
German FineWeb	43.5%
English FineWeb-Edu	39.0%
German FineWiki	12.5%
English FineWiki	5.0%

This corresponds to approximately 56% German / 44% English.

Optimization

Setting	Value
Optimizer	AdamW
β1 / β2	0.9 / 0.95
Weight decay	0.1
Gradient clipping	1.0
Effective batch	~131,072 tokens / optimizer step
Training precision	bfloat16
Framework	PyTorch + Transformers
Attention	SDPA / AOTriton on ROCm
torch.compile	Disabled
Hardware	1× AMD Radeon RX 9070 XT 16 GB
Platform	Windows + ROCm

Final held-out evaluation loss after Phase 3 was approximately 2.825 on the fixed bilingual evaluation split.

Tokenizer

Hanse2 uses a custom 32,000-token byte-level BPE tokenizer trained on a German–English mixture.

Several tokens were reserved for later instruction/tool post-training. They were not used as a chat format during base pretraining, and this checkpoint should not be prompted with an assumed chat template.

Evaluation

The following model-comparison results were produced with the EleutherAI LM Evaluation Harness using the same local setup for Hanse2-100M and Supra-50M:

• 0-shot
• no chat template
• bfloat16
• identical task implementations
• identical random seeds

Benchmark	Metric	Hanse2-100M	Supra-50M
ARC Easy	acc_norm	0.4306	0.4609
ARC Challenge	acc_norm	0.2415	0.2534
HellaSwag	acc_norm	0.3122	0.3171
WinoGrande	acc	0.4980	0.5107
PIQA	acc_norm	0.6083	0.6219
OpenBookQA	acc_norm	0.2960	0.3080
BoolQ	acc	0.5966	0.5294
SciQ	acc_norm	0.6380	0.6770
BLiMP	acc	0.8003	0.7778
MultiBLiMP German	acc	0.9852	0.7698
MultiBLiMP German	acc_norm	0.9774	0.6710

Hanse2 remains close to Supra-50M on several English benchmarks while substantially improving the German linguistic evaluation.

Long-context diagnostic

The model is configured for 8,192 tokens and produces finite forward passes at ~7.7K tokens. This does not imply reliable retrieval across the full context window.

A small base-LM continuation test placed a one-token identifier earlier in a synthetic context and measured its next-token rank against nine decoys.

Context	Needle position	Accuracy	Mean rank	Mean margin
512	10%	1.00	1.00	10.08
512	50%	1.00	1.00	10.35
512	90%	1.00	1.00	9.31
1,024	10%	1.00	1.00	9.82
1,024	50%	1.00	1.00	10.68
1,024	90%	1.00	1.00	8.14
2,048	10%	1.00	1.00	8.11
2,048	50%	1.00	1.00	10.19
2,048	90%	1.00	1.00	11.40
4,096	10%	0.20	6.70	-4.22
4,096	50%	1.00	1.00	10.34
4,096	90%	1.00	1.00	11.35
7,680	10%	0.00	5.90	-2.58
7,680	50%	0.00	5.40	-3.46
7,680	90%	1.00	1.00	11.13

The model can process long inputs, but retrieval degrades when relevant information is several thousand tokens away from the prediction point. Treat this as a small synthetic diagnostic, not a standardized long-context benchmark.

Quickstart

Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Evicka/Hanse2-100M-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

prompt = "Artificial intelligence is"
inputs = tokenizer(
    prompt,
    return_tensors="pt",
    return_token_type_ids=False,
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=150,
        do_sample=True,
        temperature=0.5,
        top_k=25,
        top_p=0.9,
        repetition_penalty=1.2,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

This is a completion model. Prompts such as sentence beginnings, paragraphs, or documents to continue are more appropriate than chat-style system/user/assistant messages.

Reproducibility

The repository includes the core scripts needed to reproduce or adapt the project:

File	Purpose
train_tokenizer.py	Train the custom byte-level BPE tokenizer
train.py	Run the three-phase pretraining pipeline
inference.py	Minimal local text generation

The training pipeline includes exact token-budget accounting, resumable checkpoints, deterministic held-out evaluation, and separate context/batch settings for each phase.

The initial training/tokenizer code was based on the Apache-2.0-licensed Supra-50M training scripts and was substantially rewritten and extended for Hanse2. No Supra model weights were used to initialize Hanse2.

Re-running the scripts does not by itself guarantee bit-identical reproduction across different hardware, driver, PyTorch, ROCm, dataset revisions, or nondeterministic kernels.

Intended use

Hanse2-100M-Base is intended primarily for:

• research on small language models
• German–English pretraining experiments
• bilingual data-mixture studies
• tokenizer and architecture experiments
• progressive context-extension experiments
• benchmarking and ablations
• continued pretraining
• supervised fine-tuning / post-training research
• education and consumer-hardware experimentation

Limitations

At ~99M parameters, Hanse2 is a small research model. Expect:

• weak factual recall and hallucinations
• weak arithmetic and multi-step reasoning
• repetition or topic drift
• brittle knowledge of specific entities
• limited long-range coherence and retrieval
• unstable behavior outside its training distribution

The checkpoint has no instruction tuning, preference tuning, RLHF, refusal training, or production safety guardrails. It may generate inaccurate, biased, offensive, harmful, or otherwise undesirable continuations.

The configured 8K context length should not be interpreted as reliable 8K retrieval capability; see the long-context diagnostic above.

License

Released under the Apache License 2.0. See LICENSE.

Acknowledgements

• Hugging Face — Transformers, Datasets, Tokenizers, FineWeb, FineWeb-Edu, and FineWiki
• EleutherAI — LM Evaluation Harness
• SupraLabs — original Apache-2.0 Supra-50M training/tokenizer scripts used as the starting point for the project code

Citation

If you use Hanse2-100M-Base in a project, a link to the model repository is appreciated.

@misc{brauer2026hanse2base100m,
  author = {Erik Brauer},
  title  = {Hanse2-100M-Base},
  year   = {2026},
  url    = {https://huggingface.co/Evicka/Hanse2-100M-Base}
}