QwenMix-3.7

QwenMix-3.7 is a weight-interpolation merge of Qwen3.6-27B and Qwen3.8-27B (Qwen3_5ForConditionalGeneration, 27B dense, hybrid linear/full attention + vision tower), produced at t = 0.5 with per-row norm preservation.

It is a merge, not a fine-tune: no gradient updates, no data. Every weight is a deterministic function of the two parent checkpoints.

Parents	Qwen3.6-27B (A, t=0) and Qwen3.8-27B (B, t=1)
Recipe	W = A + t·(B − A) per row, rescaled to t·‖B‖ + (1−t)·‖A‖
t	0.5 (global); vision tower + MTP head copied from B (t=1.0)
Math	fp32 throughout, stored bf16
Params	27.36 B (27B dense + vision), 1199 tensors
Size	55.6 GB (bf16)
Tokenizer / chat template	from Qwen3.8-27B (superset: adds 7 special tokens 248070–248076 + newer reasoning template)
License	Apache-2.0 (both parents)
Requires	transformers >= 5.x with native qwen3_5 support

Support More of This Nonsense

I publish all sorts of garage-inventor and local model stuff @ https://medium.com/@bigattichouse/ - Come read my articles and help support this sort of silliness.

What was done

1. Drift analysis. Every tensor pair (A, B) was compared. Measured cosine similarity: MLPs moved most (avg cos 0.83), linear-attention projections 0.89, full attention 0.94, SSM parameters (A_log, dt_bias, conv1d) and all layernorms 0.998–0.999 (nearly frozen), vision tower 0.993 (frozen in both parents).
2. Merge. Per-row (per-neuron) combination: take the lerp direction, then rescale each row's magnitude to the linear blend of the parents' magnitudes. This kills the norm contraction plain lerp causes between distant checkpoints (measured post-merge row-norm ratio vs A: 1.00–1.03 everywhere, i.e. no contraction).
3. Special handling.
   • Vision tower + MTP (multi-token-prediction) head: copied verbatim from B — they are identical up to rounding in both parents and must stay consistent with B's template.
   • 7 token IDs new to B's tokenizer (248070–248076): their embedding/lm_head rows were measured in A first and found to be well-trained there already (cos 0.99–1.0), but are still forced to B's values as insurance.
4. Verification.
   • Name sets, shapes, dtypes identical across A / B / merged (1199 tensors).
   • 43-tensor spot-check against an independent recomputation: bf16-rounding-level error only.
   • Float64 recheck of the two 1.27 B-element tensors: embed cos(A,W)=0.969, lm_head cos(A,W)=0.981 — both equidistant-ish as t=0.5 dictates.
   • Loads and generates cleanly with stock transformers (no trust-remote-code monkeypatching needed).

Proof that 3.7 is numerically distinct from both parents

Computed over all 1199 tensors (fp32; per-tensor data in distinctness_check.json):

Metric	Value	Meaning
mean cos(W, A)	0.9873	W's alignment with 3.6
mean cos(W, B)	0.9910	W's alignment with 3.8
mean cos(A, B)	0.9450	the parents' alignment with each other
mean ‖W−A‖/‖A‖	0.137	relative distance from 3.6
mean ‖W−B‖/‖B‖	0.111	relative distance from 3.8
median ‖W−A‖/‖W−B‖	1.06 (IQR 1.02–1.07)	≈1 → W sits at the midpoint, not skewed to one parent
merged tensors lying strictly on the A–B segment	840 / 851	cos(W,A) > cos(A,B) AND cos(W,B) > cos(A,B)
fraction of rows changed vs A / vs B	87% / 66%	the model is new, not a copy of either

Reading the numbers: W is closer to each parent than the parents are to each other (0.987/0.991 vs 0.945) and sits essentially at the midpoint (distance ratio ≈ 1.06). That is the geometric signature of a genuine t≈0.5 interpolation — not a re-skin of Qwen3.6, not a re-skin of Qwen3.8.

The 11 tensors off the segment are the vision/MTP copies (pinned to B by design) plus a handful of SSM tensors where the parents are already near-identical (cos ≈ 0.999), so the midpoint is trivially close to both. Per-group breakdown: the 851 genuinely-merged tensors have zero collapse onto either parent; the 58 SSM tensors that register as "close to A" are close to B just as much (57) — that is what a midpoint of two nearly-identical tensors looks like, not drift.

Inference: side-by-side eval (3.7 vs 3.6 vs 3.8)

Identical setup for all three: greedy decoding (temperature=0), max_new_tokens=1500, CPU bf16, default chat template (reasoning_effort=xhigh). Full raw logs in evals/.

Prompt: "In one short paragraph, what is the main idea of 'The Little Prince'?"

QwenMix-3.7 (merged) — 233 tokens

thinking: The user wants a one-paragraph summary of the main idea of "The Little Prince." I need to identify the core themes: innocence, perspective, human relationships, and the idea that "what is essential is invisible to the eye." … The paragraph should synthesize the journey, the encounters, and the ultimate moral lesson about love, responsibility, and seeing with the heart.

The main idea of The Little Prince is that the most important truths in life—such as love, friendship, and the meaning of existence—are invisible to the eye and can only be understood through the heart. Through the innocent perspective of a young prince traveling from planet to planet, Antoine de Saint-Exupéry critiques the superficiality and materialism of the adult world, which becomes obsessed with numbers, power, and possessions. Ultimately, the story teaches that what gives life value is the time and care we invest in our relationships, encapsulated in the famous lesson that "one sees clearly only with the heart," and that we are forever responsible for those we have tamed.

Qwen3.6-27B (parent A) — 897 tokens

thinking: (mechanical, multi-stage) 1. Deconstruct the user's request … 2. Brainstorm core themes … 3. Synthesize the main idea … 4. Draft the paragraph … 5. Review and Refine (Constraint Check) … 6. Final Polish …

The central theme of The Little Prince is that the most essential aspects of life—love, friendship, and imagination—are invisible to the eye and can only be perceived through the heart. Through the Prince's interplanetary journey and his poignant lessons from a fox and a rose, the story critiques the narrow, materialistic preoccupations of adulthood, which often obscure what truly matters. Ultimately, it champions the wisdom of childhood, emphasizing that meaning is found not in numbers or status, but in the unique bonds we forge and the responsibility we bear for those we "tame."

Qwen3.8-27B (parent B) — 122 tokens

thinking: We need to answer user's request … Need produce final one short paragraph. Need likely concise. Ensure one paragraph.

The main idea of The Little Prince is that what is most important in life—love, friendship, responsibility, and meaning—cannot be seen with the eyes alone, but must be understood with the heart. Through the prince's journey among adults and planets, Saint-Exupéry suggests that childhood wonder, imagination, and human connection reveal deeper truths than logic or material success.

Observations: all three are accurate and on-theme. The merge is a genuine blend — its answer depth tracks both parents, its thinking style is fluent (3.8-like) rather than the mechanical template of 3.6, and its verbosity (233 tok) lands between the verbose 3.6 (897) and terse 3.8 (122). This is a single-prompt smoke eval, not a benchmark.

Merge log

The full deterministic merge run (33.5 min, per-tensor cos + post-merge norm ratio, per-group summary, farthest-tensor list) is included as merge_log.txt. Excerpt:

[merge] 1199 tensors, A=Qwen3.6-27B B=Qwen3.8-27B t=0.5 mode=lerp-rescale group_t={'vision': 1.0, 'mtp': 1.0} dry_run=False
[merge] reading headers...
[merge] header check OK: 1199 tensors, all shapes+dtypes match
...
=== per-group summary ===
group             n      t  avg_cos(A,B)  avg_‖W‖/‖A‖
embed             1   0.50       (fp64: 0.8853)   1.03109
lm_attn          96   0.50       0.94405      1.02750
lm_head           1   0.50       (fp64: 0.9277)   1.00606
lm_lina_proj    240   0.50       0.89173      1.01953
lm_mlp          192   0.50       0.83236      1.02121
lm_norm         177   0.50       0.99817      1.00209
lm_ssm          144   0.50       0.99886      1.00227
mtp              15   1.00       0.96424      1.02220
vision          333   1.00       0.99298      1.00469

=== 10 farthest tensors (lowest cos) ===
  model.language_model.layers.0.mlp.up_proj.weight   cos=0.7139 |w/a|=1.0257
  model.language_model.layers.14.mlp.up_proj.weight  cos=0.7348 |w/a|=1.0225
  ...
[merge] done in 33.5 min

(Note: the raw log's fp32 avg_cos for embed/lm_head reads >1 — a known fp32 accumulation artifact for 1.27 B-element tensors; the true float64 values are 0.8853 / 0.9277, shown above.)

Usage

import torch
from transformers import AutoTokenizer, AutoModelForImageTextToText

model_path = "<hf-user>/QwenMix-3.7"   # this repo
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForImageTextToText.from_pretrained(
    model_path, dtype=torch.bfloat16, device_map="auto",
)

messages = [{"role": "user", "content": "Explain in two sentences why the sky is blue."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt")

out = model.generate(**inputs, max_new_tokens=1024)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Notes:

• bf16 weights need ~55 GB RAM/VRAM; use quantization (e.g. GGUF Q4_K_M) for smaller footprints.
• The chat template supports reasoning_effort (defaults to xhigh) and emits a think block before the answer. Keep max_new_tokens generous (≥1000) for complete answers.
• The vision tower is present but was never evaluated in this merge (text-only testing).

Caveats

• Merges can regress capabilities that the parents disagree on; this model was smoke-tested (single-prompt, greedy) against both parents, not benchmarked.
• Expect behavior between the parents, not strictly better than both.
• Deterministic: the same inputs + recipe always reproduce these exact weights. Per-tensor manifest: see merge_manifest.json; per-tensor distinctness data: distinctness_check.json.

Files

File	Description
model-0000*.safetensors	13 shards, bf16 weights (55.6 GB)
model.safetensors.index.json	tensor → shard map
config.json	model config (from Qwen3.8-27B)
tokenizer.json / tokenizer_config.json / merges.txt / vocab.json	tokenizer (from Qwen3.8-27B)
chat_template.jinja / generation_config.json	chat template + generation config (from Qwen3.8-27B)
preprocessor_config.json / video_preprocessor_config.json	vision preprocessors (from Qwen3.8-27B)
configuration.json	hub framework marker (Pytorch, image-text-to-text)
merge_manifest.json	merge recipe + per-group statistics
distinctness_check.json	per-tensor proof 3.7 ≠ 3.6 and ≠ 3.8
merge_log.txt	full deterministic merge run log
evals/eval_QwenMix-3.7.log	raw greedy eval output (this model)
evals/eval_Qwen3.6-27B.log	raw greedy eval output (parent A)
evals/eval_Qwen3.8-27B.log	raw greedy eval output (parent B)
LICENSE	Apache-2.0

---

Part of the QwenMix project: interpolating Qwen3.6-27B → Qwen3.8-27B. Merge engine and evaluation scripts: merge.py, verify_distinct.py, chat.py, run_eval.py (project repo).