Qwen3.5-2B-Focused-Honest

A small reasoning model that says "I can't solve this" instead of thinking until the budget runs out.

A 2B reasoning model that gets stuck on a hard problem usually doesn't give up. It recomputes, switches methods and contradicts itself until the token budget runs out, and then there is no answer at all. On the task types we tested, that happens on 68% of attempts for Qwen3.5-2B-Focused.

This model was trained to notice that it is stuck and stop with a short, honest explanation:

I attempted to calculate the duration by partitioning the span into partial month boundaries ...
However, I kept losing track of boundary inclusions and could not consistently verify my arithmetic.
UNSOLVED: Repeatedly lost track of date boundaries and leap-year tallies during manual counting.

On the same tasks, it runs out of budget on 9% of attempts. Its typical answer is twice as short.

The trade-off, stated up front: on new task types it also gives up on some problems it could have solved (see the numbers below). It does not make the model smarter, and it answers correctly somewhat less often than the base. Use it when an honest "can't" is worth more than a long, unfinished trace.

In short

All numbers: 4 independent runs (seeds) per task, 4000-token budget, answers checked by code. "New task types" are two task families the model never saw in training (80 tasks).

new task types	base (Focused)	this model
problems the base can't solve: honest UNSOLVED	0%	86%
problems the base can't solve: runs out of budget	88%	7%
problems the base can't solve: wrong answer	3%	2%
problems the base solves: correct	69%	46%
problems the base solves: gives up	0%	29%

On task types it was trained on (36 unseen tasks), it loses almost nothing on solvable problems: correct 88% → 85%. It says UNSOLVED on 96% of the unsolvable ones, where the base ran out of budget 96% of the time.

Over all 116 evaluation tasks: median tokens per attempt 4000 → 1875; correct 27% → 20%; wrong 5% → 3%; out of budget 68% → 9%.

How it was made

1. Tasks. 440 synthetic math and logic tasks in 12 families: calendar, arithmetic, modular, ordering, date_diff, base_conv, combinatorics, number_theory, sequences, logic_grid, word_rate and simulation. The tasks were written by Claude Haiku and the answers were computed by code. Two families, logic_grid and word_rate, were held out completely for evaluation. 10% of the remaining tasks were held out as validation tasks.
2. The model's own attempts. The base solved every task 4 times. Each task got a label: can't solve (0 of 4 correct), solves (3 or 4 of 4) or borderline.
3. Checking where it is really stuck. "Looks stuck" is not the same as "is stuck". From each candidate stopping point we let the base continue 8 more times, with up to 3000 tokens each.
   • If none of the 8 continuations reached the correct answer, the point is a verified dead end.
   • If at least one did, the point only looks bad. Many solvable traces look like this halfway through.
4. Training examples (757):
   • Solve, 304 examples. The model's own correct trace, cut at 1000–3000 tokens, then continued to the end.
   • Keep going, 179 examples. A point that looks stuck but where a continuation succeeded, followed by that successful continuation.
   • Give up, 274 examples. A verified dead end, cut after 2500–3000 tokens, then a short ending: a wrap-up of what went wrong plus UNSOLVED: <reason>. The endings were written by a teacher model (Gemini 3.8 Flash) that did not see the answer. It was told the problem is well-posed and the failure is the model's own, so the model never blames the problem. Endings that gave up even though the trace already had the right answer were filtered out.
5. Training. QLoRA (nf4) on the bf16 weights of Qwen3.5-2B-Focused: rank 16, alpha 32, on layers 8–23 (attention, linear-attention and MLP projections). One epoch, learning rate 1e-4 with cosine schedule, batch 8 by accumulation. The loss is 0.5 on the ending and 0.5 on the reasoning before the cut. About 10 minutes on one L40S. The adapter was then merged and quantized to Q8_0.

The format instruction in the task prompts lists ANSWER:, CONTRADICTION: and INSUFFICIENT:. UNSOLVED: is never mentioned in the prompt: the model learned to use it on its own.

Results in detail

Held-out families (logic_grid, word_rate), 80 tasks × 4 seeds. Groups follow the base model's own success rate on that task (see "How it was made", step 2).

group	n tasks	model	correct	UNSOLVED	wrong	out of budget
base can't solve	39	base	10%	0%	3%	88%
		this model	5%	86%	2%	7%
borderline	24	base	24%	0%	8%	68%
		this model	16%	64%	5%	16%
base solves	17	base	69%	0%	9%	22%
		this model	46%	29%	6%	19%

The base gets 10% of its "can't solve" tasks right here, because the labels come from 4 earlier samples and new samples regress toward the mean. Compare each row with the base row next to it, not with the label. 95% intervals, bootstrapped over tasks: this model's UNSOLVED rate on base-can't-solve tasks is 78–92%. Its correct rate on base-solves tasks is 35–56%.

Validation tasks: unseen tasks from the 10 trained families, 36 tasks × 4 seeds.

group	n tasks	base: correct / out of budget	this model: correct / UNSOLVED / wrong / out of budget
base can't solve	24	1% / 96%	1% / 96% / 3% / 0%
base solves	10	88% / 10%	85% / 10% / 0% / 5%

Other versions from the same project, on the same held-out tasks. They show the trade-off space:

	UNSOLVED on can't-solve	out of budget on can't-solve	correct on base-solves	gives up on base-solves
v2: 304 solve + 152 give-up, stopping points not verified	65%	9%	43%	40%
v3: + 179 keep-going, 152 verified dead ends	51%	38%	59%	13%
this model (v3b): + 179 keep-going, all 274 verified dead ends	86%	7%	46%	29%

Files:

• eval_tasks.json: the 116 evaluation tasks with the gold answer, split and base label.
• eval_outputs_v3b.jsonl and eval_outputs_base.jsonl: every generated text with its outcome and token count.
• eval_summary.json: outcome counts for all four models.

Limitations

• It gives up on some solvable problems of new types: 29% of attempts on held-out tasks the base solves. On trained task types this is much smaller, 10%. More "keep going" examples should reduce it, but that is untested.
• The text before UNSOLVED: is a plausible summary written in the teacher's style. It is not a verified diagnosis of what went wrong.
• Evaluated only on synthetic math and logic tasks: 116 tasks, two held-out families. 425 of the 440 tasks are in English, 15 in Russian. No standard benchmarks.
• Evaluated with base + LoRA in llama.cpp (CUDA). The merged Q8_0 file was checked against it on 12 evaluation outputs: perplexity 2.6028 vs 2.6031, compared with 3.04 for the base. A local run of the Q8_0 file on 8 validation tasks (one sample each, Vulkan) wrote UNSOLVED on all 4 unsolvable tasks and on 3 of the 4 solvable ones. In the cloud runs those 3 were solved in 1, 2 and 3 of 4 runs. One sample per task is consistent with that, but it is a small check.
• The training endings come from Gemini 3.8 Flash. The training data is not included.

Usage

llama-cli -m Qwen3.5-2B-Focused-Honest-Q8_0.gguf --temp 1.0 --top-p 0.95 --top-k 20 --presence-penalty 1.5

The evaluation used llama.cpp with exactly these sampling settings. The presence penalty was applied over the last 64 tokens, the llama.cpp default.

The same weights in other forms:

• transformers/: merged bf16 safetensors with config and tokenizer, architecture Qwen3_5ForCausalLM.
• Qwen3.5-2B-Focused-Honest-LoRA-F32.gguf: the adapter alone, for llama.cpp on top of Qwen3.5-2B-Focused-Q8_0.gguf (--lora).

Checksums: SHA256SUMS.

Lineage and license

Qwen/Qwen3.5-2B → Heretic abliteration by saidutta69 (Qwen3.5-2B-heretic) → Qwen3.5-2B-Focused, a weight edit that removes needless re-checking → this LoRA distillation, merged. Apache-2.0, inherited from Qwen3.5-2B.