Laguna-XS-2.1-DSpark

DSpark speculator for poolside/Laguna-XS-2.1. Warm-started from poolside/Laguna-XS-2.1-DFlash DFlash backbone, augmented with Eagle3-style target hidden-state conditioning, Markov acceptance head, and confidence predictor.

Built on DeepSpec (feature/laguna-dspark). DSpark = DFlash block diffusion + Eagle3 feature conditioning + Markov confidence scheduling.

Architecture


Draft Model	`LagunaDSparkModel` — 5 Llama-style layers
Hidden Size	2048
Attention	64 query / 8 KV heads, per-head softplus gate, head_dim 128
Sliding Window	512
Proposals	Up to 15 tokens per step (block_size=16)
Target Features	Layers `[1, 13, 25, 33, 39]` (39 via decoder hook for raw hidden state)
Markov Head	Rank 256, vanilla
Confidence Head	With Markov conditioning
Vocabulary	100352 (frozen target embedding + lm_head, not tied)
Precision	BF16

Weight initialization: DFlash backbone copied from Laguna-XS-2.1-DFlash. Target embed_tokens + lm_head copied frozen. Markov/confidence heads (4 tensors) randomly initialized.

Training

Hardware: Single GPU (RTX 5070, 12 GB VRAM) on WSL2 Debian. PyTorch 2.9.1+cu128, Triton 3.5.1, flex attention. NCCL/FSDP full-shard.

Data: 60 open-perfectblend conversations regenerated through BF16 Laguna target with thinking enabled. 51 passed min_loss_tokens=14 filter. Target cache: 256-token cap, raw hidden states from [1, 13, 25, 33, 39].

Hyperparameters:

Parameter	Value
Optimizer	AdamW
Learning rate	6e-4 linear warmup over 4%, cosine decay
Weight decay	0.0
Effective batch size	8 (local=1, grad accum=8)
Max steps	64
Dataset passes	11
Num anchors	8 per sample
Attention	Flex attention
Loss	`0.1CE + 0.9L1 + confidence BCE`
Loss decay gamma	4.0

Training objective (DSpark):

Cross-entropy vs target tokens (0.1 weight)
L1 distillation vs target logits (0.9 weight)
Confidence BCE for acceptance prediction (1.0 weight)

Loss curves (logged every 4 steps):

Step	CE	L1	Conf BCE	Total	Grad norm	Tau (accept len)
4	6.133	1.845	0.886	3.157	15.88	1.15
8	5.736	1.797	0.475	2.667	4.38	1.22
12	5.185	1.762	0.392	2.495	0.84	1.28
16	5.152	1.670	0.449	2.469	5.25	1.45
20	5.121	1.576	0.504	2.435	2.03	1.61
24	4.472	1.541	0.502	2.336	1.71	1.64
28	4.022	1.456	0.507	2.221	2.45	1.86
32	3.743	1.374	0.547	2.157	3.19	2.07
36	3.748	1.382	0.525	2.142	1.44	2.00
40	3.459	1.345	0.527	2.083	2.09	2.09
44	3.201	1.226	0.531	1.954	1.55	2.45
48	3.218	1.224	0.513	1.940	1.67	2.47
52	3.246	1.246	0.519	1.959	2.09	2.47
56	3.482	1.306	0.518	2.036	1.34	2.19
60	3.171	1.235	0.510	1.940	1.04	2.48
64	3.033	1.186	0.510	1.881	1.16	2.59

Expected acceptance length (tau) improved from 1.15 to 2.59. This 51-sample run demonstrates optimization but is too small for a production-quality drafter.

Full-cache deterministic evaluation (same 51 samples):

| Checkpoint | CE | L1 | Conf BCE | Total | |---|---|---|---:|---:|---:| | BF16 initialized | 7.737 | 1.900 | 0.639 | 3.124 | | Step 64 | 3.234 | 1.243 | 0.508 | 1.950 |

Save/reload preserved frozen embedding and LM-head tensors bit-for-bit.

Deployment

Pair with Laguna-XS-2.1 in vLLM or SGLang once speculative-decoding support lands (see Laguna-XS-2.1-DFlash for integration status).

vllm serve poolside/Laguna-XS-2.1 \
    --tool-call-parser poolside_v1 \
    --enable-auto-tool-choice \
    --speculative-config '{"model": "RespectMathias/Laguna-XS-2.1-DSpark", "num_speculative_tokens": 15, "method": "dspark"}'