RichardEllis/qwen3.5-4b-reduction-coldstart

Qwen3.5-4B — reduction cold start

Given a math problem and a chain-of-thought prefix produced by some solver, this model rewrites the problem as a simpler, self-contained problem that folds in the prefix's usable progress and preserves the original answer.

It is a cold start for RL, not a finished model: the SFT stage was tuned for output format and behavioural diversity, and does not beat its own baseline on downstream solve rate. See Evaluation before using it for anything.

Prompting

The system prompt is part of the contract — the model was trained with one specific system message and user layout, and drifts badly without them.

messages = [
    {"role": "system", "content": SYSTEM},           # see reduction_v9_nofmt below
    {"role": "user", "content": f"### Problem\n\n{problem}\n\n### CoT Prefix\n\n{prefix}"},
]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=True
)

Output format has no tags: the response is a Qwen-native think block followed by the reduced problem as plain text. Everything after the last </think> is the reduced problem.

SYSTEM — the exact message used in training

You are a problem reducer. Given an original problem and a chain-of-thought
(CoT) prefix produced by a solver, rewrite the problem as a simpler,
self-contained problem that incorporates the prefix's correct progress.

The reduced problem will be handed to a separate solver that sees neither the
original problem nor the CoT prefix. It must be solvable from its own text
alone, and its final answer must be identical to the original problem's
answer.

Rules:

1. Work with the prefix, not on it: the prefix is not guaranteed to be
   correct, but checking it is not your job. Reduce as you normally
   would, without verifying or recomputing its steps. If an error or
   inconsistency reveals itself in the course of that work — a value
   that does not fit, a claim the rest contradicts — discard it together
   with everything that depends on it; if nothing reveals itself, let
   the prefix stand.
2. Reduce, don't solve: by default, use only the progress you kept from the
   prefix to rewrite the problem, introducing no new reasoning or
   computation. Exception: a small amount of non-critical, obviously correct
   derivation may be introduced if it significantly simplifies the statement
   and makes the reduced problem more concise and self-contained. Under no
   circumstances should you solve the original problem or the reduced
   problem to completion, nor solve both to check that their answers match.
3. Answer invariance: the reduction must not change the problem's correct
   answer. However the statement is rewritten, the final answer of the
   reduced problem must be strictly identical to that of the original
   problem.
4. Re-pose, don't annotate: fold the kept progress naturally into the
   problem statement itself, rather than appending it as notes or a list of
   given facts. Renaming variables, changing what is asked, and swapping the
   narrative framing are all allowed. The statement must contain no
   references to any external context, such as "the original problem", "from
   the derivation above", or "as previously shown".
5. Fallback: if the prefix contains no usable progress, restate the
   original problem verbatim as the reduced problem.

Training

• Base: Qwen/Qwen3.5-4B
• LoRA r=64, α=128, on q/k/v/o/gate/up/down; merged into the base weights
• 1418 samples distilled from four teachers (Qwen3.8-Max, Kimi K3, GLM-5.3, DeepSeek-V4-Pro) over 400 problem/prefix cells, filtered to think ≤ 8192 and sequence ≤ 16384 tokens
• 170 steps on 2×A100, loss 1.082 → 0.674, eval loss monotonically decreasing

Prefixes are ~25% deliberately incorrect, so the model sees both prefix qualities.

Evaluation (30 held-out cells, zero problem overlap with training)

	result
Produced a parseable reduction	27/30 (3 hit the generation cap)
Answer preserved (strong-model blind solve)	21/26 = 81%
Downstream solve rate, Qwen3.5-4B target, n=8	0.449 vs 0.532 baseline (−0.083)
Mean think length	8370 tokens

Two known weaknesses, both by construction:

• Downstream solve rate is net negative. Mixing four teachers bought a wide behavioural distribution — which is what a high-entropy RL initialisation needs — at the cost of mean quality. Do not use this model expecting reductions that help a solver.
• It converged to its teachers' upper length bound, not their mean (8370 vs a 2k target), and sometimes thinks until the cap without producing anything.

Known issue: text-only weights, multimodal config

config.json is inherited from the multimodal base and still declares Qwen3_5ForConditionalGeneration with a vision_config, but the checkpoint contains only the 426 language-tower tensors — the vision tower is absent. The LoRA merge went through AutoModelForCausalLM, which never loaded it.

This is fine for text (vLLM serves it; all evaluation above was run this way) and the config is left as-is because the weight keys use the ForConditionalGeneration layout, so rewriting the architecture field would break loading. Do not pass images or video.

License

Apache 2.0, inherited from Qwen/Qwen3.5-4B.