Qwen3.5-35B-A3B-MXFP4-CT-RTN-AutoRound

Model Details

This is a MXFP4 quantization of Qwen/Qwen3.5-35B-A3B, generated by AutoRound. Please follow the license of the original model.

Quantization Details

• Scheme: MXFP4
• Method: AutoRound
• Base model: Qwen/Qwen3.5-35B-A3B

Evaluation

Benchmark	Score
gsm8k	0.9606
hellaswag	0.6134
mmlu	0.8240
piqa	0.8237

Reproduce & Environment

Quantized with:

auto-round --model_name /models/Qwen3.5-35B-A3B --scheme MXFP4 --model_free --ignore_layers lm_head,embed_tokens,visual,mtp,in_proj_a,in_proj_b,conv1d,shared_expert,shared_expert_gate,linear_attn,self_attn --format llm_compressor --output_dir results/qwen3_5_35b_a3b/mxfp4_explore_A/quantized

Evaluated with:

lm_eval --model vllm --model_args pretrained=results/qwen3_5_35b_a3b/mxfp4_explore_A/quantized,tensor_parallel_size=1,max_model_len=8192,gpu_memory_utilization=0.3,dtype=bfloat16,trust_remote_code=True,add_bos_token=True,enable_prefix_caching=False,max_gen_toks=2048,max_num_batched_tokens=32768,max_num_seqs=128,enable_thinking=False,reasoning_parser=qwen3,safetensors_load_strategy=prefetch --tasks gsm8k --batch_size 64 --output_path results/qwen3_5_35b_a3b/mxfp4_explore_A/eval/gsm8k --log_samples --seed 42 --num_fewshot 5 --apply_chat_template --fewshot_as_multiturn
lm_eval --model vllm --model_args pretrained=results/qwen3_5_35b_a3b/mxfp4_explore_A/quantized,tensor_parallel_size=1,max_model_len=8192,gpu_memory_utilization=0.3,dtype=bfloat16,trust_remote_code=True,add_bos_token=True,enable_prefix_caching=False,max_gen_toks=2048,max_num_batched_tokens=32768,max_num_seqs=128,enable_thinking=False,reasoning_parser=qwen3,safetensors_load_strategy=prefetch --tasks piqa,mmlu,hellaswag --batch_size 64 --output_path results/qwen3_5_35b_a3b/mxfp4_explore_A/eval/piqa_mmlu_hellaswag --log_samples --seed 42

Package	Version
auto-round	0.15.0.dev84+gd6ead479
lm-eval	0.4.12
transformers	5.16.1
torch	2.13.0+cu130
vllm	0.28.1rc1.dev312+g41848caa6.precompiled

Inference: load this repo with a vllm build that supports the MXFP4 format (day-0 architectures may need the model card's custom image/branch — check the quantize command above for the environment used).

Produced with autoquant-agent — agent-driven quantize + evaluate + self-heal.