running qwen3.5-2b in bitsandbytes nf4 precision mode really kind of destroyed it so I loaded up 3.8-27b also in nf4 and trained a rank-32 lora with a top-16k logit distillation + mse on a translation (ln->linear, safetensors for this translator included in translator.safetensors) between exact middle hidden states from 2b-nf4 to 27b-nf4 on docs from The Pile, python from The Stack and some qwen distillation datasets. idk what benchmarks look like but this is what im currently using as a prior for pg-boosted funsearch style flows on the teeniest tiniest gpu footprints. intended for a shotgun blast of few-turn rollouts.
minimal usage
from transformers import Qwen3_5ForConditionalGeneration, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
model_id = "Qwen/Qwen3.5-2B"
device = "cuda:0"
dtype = torch.bfloat16
nf4_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True
)
tok = AutoTokenizer.from_pretrained(model_id)
model = Qwen3_5ForConditionalGeneration.from_pretrained(model_id, device_map=device, dtype=dtype, quantization_config=nf4_config)
model = PeftModel.from_pretrained(model, "crumb/Q3.5-2B-4Bit-Recovery")
inputs = tok("<|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\n<think>", return_tensors="pt")
inputs = {k:v.to(device) for k,v in inputs.items()}
outputs = model.generate(**inputs, do_sample=True, temperature=1.0, max_new_tokens=256, top_k=0, cache_implementation="static")
# did you know top_k is default 50? screw that https://huggingface.co/docs/transformers/v5.20.0/en/main_classes/text_generation#transformers.GenerationConfig.top_k
print(tokenizer.decode(outputs[0]))