Teutonic-1-Chat-Preview

CortexLM Teutonic-1-Chat-Preview is a hybrid linear+softmax ~9B-class chat model with thinking-style reasoning, trained for long-context conversation and verifiable math/code problem solving.

Preview — still under active training. Expect weight updates, better alignment, and longer reliable generations. This card is a snapshot, not a final release.

Highlights

• Hybrid architecture (linear attention + softmax attention) in the Qwen3.5 text lineage
• Default weights: FP8 (block-scaled e4m3, Qwen-style quantization_config) for efficient inference
• RoPE context capacity up to 256k input tokens
• Chat template with optional <think>…</think> reasoning blocks and tool-calling tags
• Trained with continued long-context mid-training, supervised chat+thinking+tools, and RL on verifiable rewards

What’s done vs what’s still cooking

Done in this preview	Still cooking
Long-context continued pretraining (up to 64k packs)	More preference / alignment optimization
Supervised chat, thinking, and tools	Broader RL beyond math/code verifiers
RL on verifiable math (and code-oriented) rewards	Reliable 64k free-form generation (in progress)
Long-output continuation (16k→32k response budgets)	Multimodal later
256k input continued training (experimental; see below)	Merge long-ctx without math regression
256k RoPE capacity retained on default weights	Multikey / hard long-ctx retrieval

Practical limits today: default weights keep strong chat/math quality with RoPE capacity 256k. A separate 256k-input continued-training checkpoint exists (needle retrieval strong) but is not the default download yet — math quality dropped after that pass. Generation defaults target long answers (up to 64k in generation_config), but stable 64k-out training is still underway.

How it was trained (high level)

1. Mid-training — continued pretraining with a long-context curriculum (8k → 32k → 64k packs) to extend useful context under FP8 training.
2. Supervised chat fine-tune — chat, thinking traces, and tool-use style data.
3. Reinforcement learning — policy optimization against verifiable math/code rewards (boxed-answer style grading).
4. Long-context / long-output continuation — further supervised + RL passes to keep needle-style long-context behavior and raise response-length budgets.

Training charts (real logged metrics)

Overview of mid-training, supervised chat fine-tune, RL verifiable rewards, and long-output continuation.

Mid-training train loss (smoothed) with held-out val points across the long-context curriculum.

Supervised chat+thinking fine-tune loss, plus a later long-context SFT continuation.

RL mean verifiable reward and response clip ratio; lower panel shows long-output continuation runs.

Quick start

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "CortexLM/Teutonic-1-Chat-Preview"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    device_map="auto",
    torch_dtype="auto",  # loads FP8 weights via quantization_config
)

messages = [
    {"role": "system", "content": "You are Teutonic, a helpful reasoning assistant."},
    {"role": "user", "content": "Solve: 17 * 19. Put the final answer in \\boxed{}."},
]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,  # if supported by the template path you use
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(
    **inputs,
    max_new_tokens=4096,
    temperature=0.6,
    top_p=0.95,
    do_sample=True,
)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=False))

Thinking content appears inside <think>…</think> when the model opts to reason before the final answer. Turns end with <|im_end|>.

vLLM

vllm serve CortexLM/Teutonic-1-Chat-Preview \
  --trust-remote-code \
  --dtype auto \
  --max-model-len 32768

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="CortexLM/Teutonic-1-Chat-Preview",
    messages=[{"role": "user", "content": "Write a short haiku about glaciers."}],
    temperature=0.7,
    top_p=0.9,
    max_tokens=512,
)
print(resp.choices[0].message.content)

Recommended sampling

Mode	temperature	top_p	notes
Reasoning / math	0.5–0.7	0.90–0.95	Prefer \boxed{} for graded answers
Chat / creative	0.7–0.9	0.90–0.95	—
Deterministic smoke	0.0	—	do_sample=False

Context & generation

• Input capacity (config): up to 256k tokens (RoPE max_position_embeddings=262144)
• Generation target: moving toward 64k free-form outputs; this preview is strongest at moderate-to-long answers (multi-k tokens). Treat 64k-out as aspirational until a later drop.

Long-context continued training (256k)

We completed an experimental 256k-input continued-training curriculum (128k → 256k packs) after the chat/RL path. Results:

Capability	Status
Single-needle retrieval @64k / 128k / 256k	Strong (pass rate 1.0 in internal needle smokes)
Multi-key needle @128k	Weak (~0.4) — follow-up recommended
Chat behaviour / tools format smoke	Holds (10/10 internal cases)
Verifiable math (internal private probe)	Regressed vs the default chat checkpoint — therefore not promoted to default weights
64k free-form output training	Still in progress (orthogonal track)

Default weights on this repo remain the stronger chat + math snapshot (FP8). The 256k-in experimental checkpoint lives in our training volume as teutonic-ii-lc256k (alias teutonic-ii-longctx-256k); a sibling 128k checkpoint is teutonic-ii-lc128k. We will publish a dedicated long-context weight drop only after math recovers (co-training / replay mix).

Evaluation honesty

Internal math/reasoning probes improved substantially after verifiable-reward RL relative to the supervised merge. We do not claim contaminated public-bench SOTA (e.g. treat GSM8K-style numbers with caution). Prefer private or carefully decontaminated suites when comparing. Behavioural chat smokes (format, tools, basic instruction following) pass on this snapshot; broader alignment and preference quality are still in progress.

Limitations

• Preview checkpoint — APIs and weights will change
• Long free-form generation beyond ~32k tokens is not fully hardened
• Not fully preference-aligned; may over-think or under-refuse
• Hybrid kernels need recent transformers / vLLM with trust_remote_code
• FP8 inference quality tracks the Qwen-style block FP8 path; if your stack lacks FP8 support, request a BF16 variant or dequantize

License & acknowledgements

Released under Apache 2.0.

This model continues the Qwen3.5 text architecture lineage (hybrid Teutonic conversion). Please respect the Qwen / Alibaba base model terms and citations where applicable:

• Qwen
• CortexLM / Dendrite training stack

Citation

@misc{teutonic1chatpreview2026,
  title        = {Teutonic-1-Chat-Preview},
  author       = {CortexLM},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/CortexLM/Teutonic-1-Chat-Preview}},
  note         = {Preview checkpoint; under active training}
}

Benchmarks

See benchmarks.md for the corrected same-harness comparison vs Qwen3.5-9B (chat + thinking on, fixed 2026-09-01). The first published Qwen column (MMLU-Pro ~21 / MATH ~20) was an invalid harness artifact (thinking ON + wrong chat template + short max_tokens → mid-think truncation).

Benchmark	Teutonic	Qwen3.5-9B (fixed harness)	Δ
MMLU-Pro (n=2000)	69.8	73.2	−3.4
MATH-500	88.6	68.2	+20.4
HumanEval+	82.9	81.1	+1.8
MBPP+	75.1	64.6	+10.5
IFEval	68.6	65.4	+3.2
RULER (4k–64k)	89.5	88.4	+1.1

Protocol: chat · thinking on · raised think budgets · seed 20260830. Vendor card numbers are separate; see benchmarks.md.