Qwen3.8-Flash-Next-MXFP4-Mixed-FP8KV-CT-RTN-AutoRound

Model Details

This is a MXFP4-Mixed quantization of Qwen/Qwen3.8-Flash-Next, generated by AutoRound. Please follow the license of the original model.

Quantization Details

• Scheme: MXFP4-Mixed
• Method: AutoRound
• Base model: Qwen/Qwen3.8-Flash-Next

Evaluation

Benchmark	Score
gsm8k	0.9280
hellaswag	0.6781
mmlu	0.8646
piqa	0.8270

Reproduce & Environment

Quantized with:

auto-round --model_name /models/lkk/cri_models/Qwen3.8-Flash-Next --scheme MXFP8 --model_free --nsamples 32 --static_kv_dtype fp8 --ignore_layers visual,lm_head,embed_tokens,mlp.gate,mlp.shared_expert_gate,in_proj_a,in_proj_b,block_inject_weight,ple,mtp,hyper_connection --layer_config '{"mlp.experts":{"bits":4,"data_type":"mx_fp"}}' --device_map auto --format llm_compressor --output_dir /models/auto-quant/results/qwen38_flash_next/mxfp4_mixed_static_fp8kv/quantized --disable_torch_compile --low_gpu_mem_usage

Evaluated with:

lm_eval --model vllm --model_args pretrained=/models/auto-quant/results/qwen38_flash_next/mxfp4_mixed_static_fp8kv/quantized/Qwen3.8-Flash-Next-mxfp-w8g32,tensor_parallel_size=1,max_model_len=8192,gpu_memory_utilization=0.85,dtype=bfloat16,trust_remote_code=True,add_bos_token=True,enable_prefix_caching=False,max_gen_toks=2048,kv_cache_dtype=fp8,max_num_batched_tokens=16384,max_num_seqs=64,language_model_only=True,reasoning_parser=qwen3,enable_thinking=False,safetensors_load_strategy=prefetch --tasks gsm8k --batch_size 64 --output_path /models/auto-quant/results/qwen38_flash_next/mxfp4_mixed_static_fp8kv/eval/gsm8k --log_samples --seed 42 --num_fewshot 5 --apply_chat_template --fewshot_as_multiturn
lm_eval --model vllm --model_args pretrained=/models/auto-quant/results/qwen38_flash_next/mxfp4_mixed_static_fp8kv/quantized/Qwen3.8-Flash-Next-mxfp-w8g32,tensor_parallel_size=1,max_model_len=8192,gpu_memory_utilization=0.85,dtype=bfloat16,trust_remote_code=True,add_bos_token=True,enable_prefix_caching=False,max_gen_toks=2048,kv_cache_dtype=fp8,max_num_batched_tokens=16384,max_num_seqs=64,language_model_only=True,reasoning_parser=qwen3,enable_thinking=False,safetensors_load_strategy=prefetch --tasks piqa,mmlu,hellaswag --batch_size 32 --output_path /models/auto-quant/results/qwen38_flash_next/mxfp4_mixed_static_fp8kv/eval/piqa_mmlu_hellaswag --log_samples --seed 42

Package	Version
auto-round	0.16.0.dev155+g51003909
lm-eval	0.4.13
transformers	5.17.0
torch	2.13.0+cu130
vllm	0.29.1rc1.dev528+g9815732c1

Inference: load this repo with a vllm build that supports the MXFP4-Mixed format (day-0 architectures may need the model card's custom image/branch — check the quantize command above for the environment used).

Produced with autoquant-agent — agent-driven quantize + evaluate + self-heal.