XT-Lunaris-4B
DRAFT CARD, NO WEIGHTS YET. Training is paused at step 150 of 752 and no adapter or merged checkpoint is uploaded. This repo exists to publish the intent, the measured configuration and the evaluation protocol. Card stamped: repo version 2.4.0-Lunaris, 2026-10-26.
What this model is meant to be
XT-Lunaris-4B is a focused fine-tune of Qwen/Qwen3.5-4B into the XT assistant character:
- Long, structured thinking. Reasoning happens in the model's own output, not behind an inference-time cap. Efficiency is supposed to come from training (SFT, then RL), never from truncating reasoning at serve time.
- A humanized voice. Conversational, plain, no boilerplate hedging, no corporate cadence.
- Confirm before destructive actions. A trained and gated behaviour: the model asks before anything irreversible, and the gate is measured, not assumed.
Base model
Qwen/Qwen3.5-4B (Apache-2.0), 4.21B parameters, 32 layers, 248,320 token
vocabulary, hybrid Gated DeltaNet plus gated attention layout, 262,144 native
context.
Training data
XtrCom/XT-Research-FT-Dataset (CC-BY-4.0), 50,646 rows / 48.80M measured
tokens, built from public sources with per-source attribution.
Focused slice used for the current run: focus_2k, 10,180 rows / 6.15M
tokens (code excluded, reasoning and companion weighted).
Training procedure (as actually configured)
| setting | value |
|---|---|
| method | LoRA, r=64, alpha=128, dropout 0.05, all-linear targets |
| base precision | bf16 (4-bit QLoRA was tried and rejected by measurement: activation blow-up at 4k) |
| context | 2048 tokens |
| batching | batch_tokens 2048, grad accum 4, length-bucketed |
| optimizer | AdamW, lr 1e-4, cosine decay, 20 warmup steps, betas 0.9 / 0.95, no weight decay |
| seed | 20261017 |
| hardware | NVIDIA L4 22 GB, 15.55 GB peak allocated, ~983 tok/s |
Why there are no weights yet
The run exposed a masking defect: with --mask-no-think, any assistant turn
that carried no think block returned an all-masked label row from an early
return and never reached the fallback, so 5,429 of 10,180 focus rows produced
zero supervised tokens. Half of the voice and prose data trained on
nothing, and the zero/0 means over those batches produced the NaN windows in
the log.
The fix masks empty think blocks only, trains ordinary no-reasoning turns normally, and drops zero-signal rows at build time with a count. Behaviour checks exist for it. The step-150 adapter survives as a baseline only and is deliberately not published, because its training signal was wrong.
Evaluation protocol
- Probe battery over voice, structure, honesty, refusal behaviour and the confirm-before-destructive policy.
- Five-model matrix that includes stock bf16 and stock Q4 controls, so any gain is measured against the base model rather than vibes.
- Capability gates must hold: no degradation on reasoning or code relative to the base.
Known limitations
- Not released. Nothing in this repo should be treated as a usable checkpoint.
- Long-context training (4k and 8k) is blocked on GPU quota: the 248k-vocabulary logits tensor alone needs chunked cross-entropy before 8k fits on 22 GB.
- Abliteration is used only as a small, capability-gated experiment. Measured result on the sibling MiniCPM5-2B study: the refusal direction is causally identified but the behaviour does not follow (benign refusals went up, p=0.006), so refusal is handled by data and RL instead.
License
Card and compilation CC-BY-4.0. The base model Qwen/Qwen3.5-4B is
Apache-2.0, and any distributed weights must comply with that license too.
Upstream dataset terms are preserved per source.