Frosty40/laguna-xs-2.1-b70-serving

Laguna XS 2.1 on Intel Arc Pro B70 — SYCL Serving Package

Serving scripts, validated configuration, and benchmark receipts for running poolside's Laguna XS 2.1 (Q4_K_M GGUF) on a single Intel Arc Pro B70 GPU with an optimized llama.cpp SYCL build.

No model weights are included. Download the official GGUF from poolside/Laguna-XS-2.1-GGUF (license: OpenMDW-1.1). The optimized kernels ship here as patches/lx-champion-vs-b10167.patch, applicable to upstream llama.cpp commit 7e1e28cae (build instructions below). The tuning harness, experiment notes, and raw benchmark receipts behind these numbers live in newjordan/lagunaX. An upstreaming effort to mainline llama.cpp (cleaned, per-feature PRs) is in progress.

Benchmark numbers (receipts included)

Serial single-stream llama-bench, Q4_K_M, measured 2026-08-09 (receipt: receipts/20260809T191208Z-*, 5 reps each):

Metric	Optimized tree	Stock baseline*	Delta
Decode (tg128)	152.55 t/s (stddev 0.06)	107.35 t/s	+42.1%
Prefill (pp512)	1170.98 t/s (stddev 9.7)	1139.15 t/s	+2.8%

*Baseline: pinned base-control build of the same-era upstream source, same GPU/model/flags, pinned 2026-07-29 (receipts/20260809T191208Z-score.json).

Claim boundary (from the receipt itself): serial one-stream pp512+tg128 only — not multi-slot aggregate tok/s, and not comparable across different quants/silicon.

Decode at KV depth (llama-bench -p 0 -n 128 -d, measured 2026-08-10)

KV depth	decode	notes
0	152.5 t/s	board receipt above
4,096	129.5 t/s ±0.3	VEC attention path (+11% vs the tile kernel)
16,384	102.7 t/s ±0.3	VEC attention path (+29% vs the tile kernel)

Real-usage receipts (llama-cli, chat template, measured 2026-08-10)

task	prompt eval	generation
short chat prompt (writing task, 700 tok gen)	95.5 t/s*	145.9 t/s
short chat prompt (coding task, 900 tok gen)	96.3 t/s*	144.8 t/s
24K-token document ingest + continue	300.5 t/s	91.2 t/s at ~24K depth

*short-prompt "prompt eval" is dominated by launch overhead, not throughput; see pp512 for sustained prefill. The 24K ingest used llama-cli defaults (-c 32768, default ubatch); the tuned server config (-ub 2048) prefills faster. Raw receipts: lagunaX repo, results/lx-usage-receipts-20260810T154850Z/ and results/lx-fattn-depth-ab-20260810T*/.

Multi-slot campaign (separate track, measured 2026-07-29, campaign notes in docs/): aggregate decode ~513–522 t/s banked at -np 64 -c 32768 -ub 1024 -b 8192 with continuous batching under synthetic load. Single-request full context serving decodes ~85 t/s at -c 131072 depth (2026-08-05 A/B, serving/ab-ubatch-laguna.sh header).

Hardware / software (as measured)

Component	Version
GPU	Intel(R) Arc(TM) Pro B70 Graphics — 256 CUs, 30.3 GiB VRAM (clinfo, 2026-08-07)
CPU	AMD Ryzen 9 5950X (16C/32T)
Kernel	Linux 7.0.0-28-generic, xe driver (Ubuntu 24.04)
GPU runtime	intel-opencl-icd / libze-intel-gpu1 26.18.38308.1, libze1 1.28.2
oneAPI	2026.0 (icpx compiler, oneDNN 2026.0, MKL 2026.0)
Model	Laguna-XS-2.1-Q4_K_M.gguf (~20.3 GB), official poolside GGUF

Build the optimized llama.cpp (SYCL)

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout 7e1e28cae
git apply /path/to/patches/lx-champion-vs-b10167.patch

source /opt/intel/oneapi/setvars.sh
cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=icpx \
  -DGGML_SYCL=ON \
  -DGGML_SYCL_F16=ON \
  -DGGML_SYCL_TARGET=INTEL
cmake --build build -j16 --target llama-server llama-bench

Mirrors the CMake cache of the receipt binary (CMAKE_BUILD_TYPE=Release, GGML_SYCL=ON, GGML_SYCL_F16=ON, GGML_SYCL_TARGET=INTEL, icpx, GGML_NATIVE=ON, GGML_OPENMP=ON).

Serve — single stream, full 131,072-token context (port 8092)

serving/serve-laguna.sh. Edit REPO (and the binary path) to point at your build from the step above, and MODEL at your GGUF, then:

./serving/serve-laguna.sh          # PORT=8092 CTX=131072 NPARALLEL=1 by default

Key flags: -ngl 99 -fa on -ctk f16 -ctv f16 -c 131072 -np 1 -b 4096 -ub 2048 --jinja --chat-template-file poolside-Laguna-XS-2.1.jinja --temp 1.0 --top-k 20 --top-p 1.0 --min-p 0.0. A copy of the chat template ships in serving/.

Bench-validated env for the B70 (from the 2026-08-09 receipt):

export ONEAPI_DEVICE_SELECTOR=level_zero:gpu
export ZE_AFFINITY_MASK=0
export GGML_SYCL_DISABLE_GRAPH=1
export GGML_SYCL_DISABLE_DNN=1
# REQUIRED for the receipt decode numbers: enables the fused RMS-norm+RoPE
# KV-cache-write kernels, which default OFF in this tree. Without it decode
# lands ~3-5% lower.
export GGML_SYCL_FUSE_NORM_ROPE=1

Two gotchas that look like "truncated output" (diagnosed 2026-08-04)

1. -c is the TOTAL KV pool split across -np slots. -c 131072 -np 4 gives each request only 32,768 tokens. Use -np 1 for the full trained context (VRAM is identical, ~25 GiB either way; 262,144 total does not fit in 30.3 GiB). Check curl :8092/props → default_generation_settings.n_ctx (per-slot, not total).
2. Laguna is a thinking model; reasoning tokens bill against the client's max_tokens. A small cap (e.g. 400) can be consumed entirely inside the reasoning block, returning empty content with finish_reason: "length". Send no max_tokens (server default n_predict: -1) or a generous one (>1500), or disable thinking per-request with "chat_template_kwargs": {"enable_thinking": false}.

Why -ub 2048 and not 4096

A/B'd at -c 131072 on 2026-08-05 (serving/ab-ubatch-laguna.sh): 4096 is slower at every depth — prefill −7.1/−6.0/−3.6% at 6.5K/26K/52K tokens, decode −10% (84.6 → 75.9 t/s) — and costs +1.3 GiB VRAM.

Serve — multi-slot throughput

serving/env.sh (shared campaign env; adjust the TB_PKG/LAGUNA_* paths to your build) plus serving/max-push-2h.sh (multi-slot llama-server + concurrent load via serving/max-push-load.py + telemetry). Campaign status and banked numbers: docs/OPERATIONAL.md, docs/CAMPAIGN-README.md.

Package contents

serving/serve-laguna.sh              single-stream :8092 launcher (full 131K ctx)
serving/env.sh                       shared B70/Laguna env (paths + GGML_SYCL_* knobs)
serving/ab-ubatch-laguna.sh          -ub 2048 vs 4096 A/B harness (receipt in header)
serving/max-push-2h.sh               multi-slot serve + load + telemetry campaign
serving/max-push-load.py             concurrent load generator
serving/poolside-Laguna-XS-2.1.jinja chat template used with --jinja
docs/                                multi-slot campaign status notes
receipts/20260809T191208Z-*          llama-bench receipt for 152.55 / 1170.98

Licensing

• Laguna XS 2.1 weights and the official GGUF: OpenMDW-1.1 (permissive open model weights license) — see the model card.
• llama.cpp and this fork: MIT (upstream llama.cpp license).
• Scripts in this package: same license as the lagunaX repository.

Note: scripts reference absolute paths from the original box (/home/frosty40/..., /mnt/data2tb/...); edit the path variables at the top of each script for your machine.