Jackrong

/

Qwopus3.8-27B-Flash

like 13

Thanks for releasing this model — the reasoning quality looks genuinely good, which is why I wanted to report this rather than just move on.

Summary

When generating Python, the model emits a single space of indentation regardless of nesting depth. Depth 1, 2 and 3 all come out as one space, so the code does not parse.

This is not a quantization or GGUF conversion issue — I checked. It reproduces when loading the safetensors directly with transformers.

Reproduction

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

MODEL = "Jackrong/Qwopus3.8-27B-Flash"
PROMPT = ("Write one Python function squeeze(items) that removes consecutive "
          "duplicates from a list of integers. Return only a python code block, "
          "no explanation. Use a loop with an if statement inside it.")

q = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16,
                       bnb_4bit_quant_type="nf4")
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, quantization_config=q,
                                             device_map="auto").eval()

text = tok.apply_chat_template([{"role": "user", "content": PROMPT}],
                               tokenize=False, add_generation_prompt=True)
inp = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**inp, max_new_tokens=256, do_sample=False,
                     pad_token_id=tok.pad_token_id or tok.eos_token_id)
print(tok.decode(out[0][inp["input_ids"].shape[-1]:], skip_special_tokens=True))

Actual output

def squeeze(items):
 result = []
 for item in items:
 if not result or result[-1] != item:
 result.append(item)
 return result

IndentationError: expected an indented block after 'for' statement

Note that the body of the for and the body of the if sit at the same indentation as the for itself, so the block structure is unrecoverable — this cannot be fixed by re-indenting afterwards.

Token-level evidence

I collected the whitespace-only tokens from the generated sequence and compared against Qwen3.8-27B running on the same llama.cpp build, same arch (qwen35), same prompt:

Qwen3.8-27B uses the depth-specific indentation tokens correctly. Qwopus-Flash never selects any of them — the leading space is merged into the following word token, so every nesting level ends up with exactly one space.

The tokenizer itself is fine. A round-trip confirms the vocabulary can represent 4-space indentation:

"def f():\n    result = []\n"
  -> 727 'def' / 281 ' f' / 4406 '():' / 198 newline / 262 (3 spaces) / 1067 ' result' / ...

What I ruled out

Impact

On my own SWE-bench-style harness — not the official SWE-bench, but 60 self-written tasks where the model patches a bug and hidden pytest decides pass/fail (zephel01/swe-bench, MIT, stdlib-only) — 14 of 60 tasks scored zero before any test ran, because the generated file could not be parsed. Only 2 tasks failed on actual logic. The reasoning in the collapsed files looks correct — it is purely the indentation that makes them unrunnable.

Question

Is this intentional? The model card mentions "9.9% fewer characters", and collapsing 4-space indentation to 1 space would be one way to achieve that. If it is a side effect of the efficiency training (or of whitespace normalization in the SFT corpus), it may be worth a note on the model card, since it makes the model unusable for code generation in its current form.

Happy to re-test if you push an updated version.

Environment

• transformers + bitsandbytes 4bit (nf4), greedy, max_new_tokens=256
• Also reproduced on llama.cpp with the official MTP-Q4_K_M and MTP-Q5_K_M GGUFs
• Control model: Qwen3.8-27B Q6_K, same llama.cpp build
• Benchmark harness: https://github.com/zephel01/swe-bench (MIT, stdlib-only, reproducible)