XT-Lunaris-1B

DRAFT CARD, NO WEIGHTS YET. This is the planned successor to XtrCom/XT-Luna-1B, rebuilt from scratch with long context as the primary axis. Card stamped: repo version 2.4.0-Lunaris, 2026-10-26.

What this model is meant to be

XT-Lunaris-1B is the second generation of XT-Luna: a small on-device assistant aimed at agentic long-horizon work in 8k to 32k contexts, keeping the XT character (long structured thinking, humanized voice, confirm before destructive actions).

Why a 1B for this: at this size the whole model plus long-context training state fits a single 22 GB L4, so 8k, 16k and 32k widths can be measured and compared instead of assumed.

Base model

openbmb/MiniCPM5-1B (Apache-2.0), verified configuration:

field	value
architecture	standard transformer (llama model type), 24 layers
hidden size	1536
vocabulary	130,560
native context	131,072 (no RoPE scaling needed for 8k to 32k)
dtype	bfloat16

Weights are trained from the base, not from the merged GGUF of the previous generation: a merged quantized model cannot be used as an adapter training target.

Why the previous generation is being redone

Measured row lengths in the existing slices are short: median 561 to 600 tokens, 90th percentile around 1,450, and the slice built specifically for context adaptation has no row above 1,882 tokens. Training at 8k or 32k on that data would be almost entirely packing of unrelated short conversations, which teaches a model to fill a context window rather than to use one.

So the first real work of this generation is the data, not the run: genuinely long rows (4k to 32k) covering long agent sessions, long code tasks, and long-form conversational turns, sourced from permissively licensed public datasets and from repositories where provenance is ours to state.

Training plan (stages, in order)

1. Long-context data build with provenance and per-source license terms.
2. Width measurement on the L4 at 8k, 16k and 32k: peak memory and tok/s for each width, with chunked cross-entropy (a 32k fp32 logits tensor would be about 17 GB on this vocabulary, so scoring must be chunked).
3. SFT at the width the measurement picks, with gradient checkpointing (already enabled in the trainer) and sequence packing as a complement to long rows, never a substitute for them.
4. OPD with a stronger teacher, candidate selection by our own probe battery rather than published tables: ornith-ai/Ornith-1.5-9B (MIT, 9B) and XHToken/Spark-X2.5-4B (Apache-2.0, 4B, 1M native context) against the Qwen3.5-4B control. Note the L4 constraint: a 9B bf16 teacher is about 19.3 GB and cannot share the card with the student, so it would run quantized or as an offline generator.
5. GRPO for the behaviours SFT can only suggest: efficiency of reasoning (no inference caps), confirm-before-destructive, refusal on genuinely harmful requests.

Evaluation

Probe battery covering voice, structure, honesty, refusal behaviour and the destructive-action gate, plus long-context specific checks: task located deep in the context, long code comprehension, and continuation of long agent sessions. Any claim of improvement is measured against the stock base model in the same harness.

Status and limitations

• No weights, no adapter, no quantization in this repo yet.
• Stage 1 and stage 2 are the current work; everything after that is planned, not true.
• Budget discipline: runs are costed before launch against a $100 campaign cap with a $5 to $10 per-session line.

License

Card and compilation CC-BY-4.0. The base model openbmb/MiniCPM5-1B is Apache-2.0, and any distributed weights must comply with that license too. Predecessor: XtrCom/XT-Luna-1B.