Luigi/qwen35-0.8b-arcsum

arcsum — Qwen3.5-0.8B agentic zh-TW meeting summariser (qwen-tools-v5)

An on-device agent that reads a meeting transcript in ~2,500-token chunks, maintains a small external memory (ARC + POINTS) across steps, and emits one flowing Traditional Chinese summary. The model never sees conversation history: each step is (system, MEMORY + CHUNK) -> tool call, and a harness applies the edits deterministically.

Trained for the single-turn tool-call protocol: each reading step emits ONE batched update_memory call with arc / add / drop arguments.

Serving — read this first

Serve with --no-jinja. Every number below was measured that way (plain ChatML, prompt ending <|im_start|>assistant\n).

llama-server -m Qwen3.5-0.8B.Q8_0.gguf --no-jinja -c 4096 -C 0xFF

Note a real inconsistency in this checkpoint: it was trained with the base repo's chat template, which appends a closed <think>\n\n</think>, but evaluated without it. The gate results below therefore describe the --no-jinja configuration. Serving under --jinja puts an open <think>\n in the prompt (a different branch again) and is NOT the configuration these numbers came from.

For reproducibility also pin cache_prompt: false — llama.cpp's prompt cache changes generation (measured: 700 vs 167 characters, same seed and temperature).

Use repeat_penalty=1.1 on the synthesis call only. Reading steps emit a fixed op vocabulary and a repetition penalty there punishes the format's own tokens.

Measured results

40 held-out MeetingBank-derived zh-TW meetings, agent vs a fair map-reduce baseline (same model, same chunk size):

gate	result
G2 faithfulness	PASS — 16 vs 58 inversions, 40/40 paired
G3 rouge1	PASS — 28/12, +0.069, p=0.017
G3 rouge2	PASS — 29/11, +0.041, p=0.006
G3 rougeL	PASS — 35/5, +0.057, p=0.000
G4 latency	PASS — 19.0 min/meeting measured on an Oppo Reno 7 (CPU-only, Q8, all 8 cores) vs a 20-minute ceiling
real zh-TW ASR	17/20 meetings curated
G1 within-meeting revision	FAIL — 5/27 on an independent 27-scenario probe

Known limitations

• Within-meeting revision fails. If a decision is taken and then reversed later in the same meeting, the summary may report the superseded decision as current. Both training corpora (MeetingBank, and 20 real zh-TW legislative meetings) contain essentially no within-meeting reversals, so this capability was never learnable from the data.
• G4 margin is thin. 19.0 min nominal, but a contended phone measured 21.6 min — over budget. Thermal throttling is not the cause (flat over 29.5 min of load); process contention is.
• G2's per-claim rate favours the baseline (5.7% vs 4.0%). The agent wins on absolute inversions partly because it asserts far fewer claims (283 vs 1,467).
• zh-TW only. English is source material, never a product language.

Provenance

Base: Qwen/Qwen3.5-0.8B (text tower only; the vision encoder is not used and is dropped at export). The MTP head is restored from the base checkpoint before GGUF conversion — llama.cpp requires those 15 tensors at load time even though training never modifies them.