amd/Qwen3.6-35B-A3B-Quark-INT4-G128

Qwen3.6-35B-A3B RTN INT4 W4A16 (experts + attn)

Quark INT4 symmetric weight-only (W4A16) quantization of Qwen/Qwen3.6-35B-A3B.

Quantization

Item	Value
Scheme	int4_wo_128 symmetric, group size 128
Method	RTN (round-to-nearest, no calibration)
Pack	Quark real_quantized, pack_method=reorder
Quantized	MoE experts + self_attn + linear_attn
FP16 kept	mlp.gate, shared expert, lm_head, vision, MTP

Evaluation (Wikitext-2 test)

Metric	Value
Perplexity	6.990
Tokens scored	296,815
Chunks (seq_len=2048)	145
Eval time	~249 s

See eval/ppl_wikitext.json for full results.

Evaluation (Open LLM Leaderboard v2, lm_eval)

Evaluated with EleutherAI/lm-evaluation-harness --tasks leaderboard via vLLM on 2026-07-22. All six leaderboard-v2 tasks were measured on this model. Run config: tensor_parallel_size=4, gpu_memory_utilization=0.90, max_model_len=8192, VLLM_USE_TRITON_AWQ=1.

Task	Metric	BF16 (base)	UINT4	INT4 (this)
IFEval	prompt_level_strict	0.3235	0.3383	0.3438
IFEval	inst_level_strict	0.4580	0.4664	0.4676
MATH-hard	exact_match	0.5287	0.4856	0.5083
BBH	acc_norm	0.6546	0.6270	0.6018
GPQA	acc_norm	0.4337	0.3658	0.3431
MMLU-Pro	acc	0.5968	0.5824	0.5687
MuSR	acc_norm	0.4339	0.4206	0.4630

Per-task raw metrics are shown (the six tasks use different metrics, so no single average is reported). Full result JSON and per-sample JSONL files are in eval/leaderboard/.

Evaluation (GSM8K, 5-shot)

Evaluated alongside the BF16 base model and the UINT4 variant under identical conditions. The prompt suffix <think>\n\n</think>\n\n is prepended to each answer field to suppress the model's thinking preamble so all three runs are comparable (see custom task yaml below).

Model	flexible-extract	strict-match
Qwen3.6-35B-A3B BF16 (base)	0.9158 ± 0.0076	0.9090 ± 0.0079
Qwen3.6-35B-A3B RTN-INT4-W4A16 (this model)	0.9045 ± 0.0081	0.8961 ± 0.0084
Qwen3.6-35B-A3B RTN-UINT4-W4A16	0.9022 ± 0.0082	0.8976 ± 0.0083

Eval command:

# 1. Create a custom task yaml that suppresses thinking (save as gsm8k_nothink/gsm8k_nothink.yaml)
cat << 'EOF' > gsm8k_nothink/gsm8k_nothink.yaml
task: gsm8k_nothink
dataset_path: openai/gsm8k
dataset_name: main
output_type: generate_until
training_split: train
fewshot_split: train
test_split: test
doc_to_text: "Question: {{question}}\nAnswer: <think>\n\n</think>\n\n"
doc_to_target: "{{answer}}"
metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true
    ignore_case: true
    ignore_punctuation: false
    regexes_to_ignore:
      - ","
      - "\\$"
      - "(?s).*#### "
      - "\\.$"
generation_kwargs:
  until:
    - "Question:"
    - "</s>"
    - "<|im_end|>"
  do_sample: false
  temperature: 0.0
repeats: 1
num_fewshot: 5
filter_list:
  - name: "strict-match"
    filter:
      - function: "regex"
        regex_pattern: "#### (\\-?[0-9\\.\\,]+)"
      - function: "take_first"
  - name: "flexible-extract"
    filter:
      - function: "regex"
        group_select: -1
        regex_pattern: "(-?[$0-9.,]{2,})|(-?[0-9]+)"
      - function: "take_first"
metadata:
  version: 3.0
EOF

# 2. Run evaluation
CUDA_VISIBLE_DEVICES=0 \
VLLM_USE_TRITON_AWQ=1 \
lm_eval run \
  --model vllm \
  --model_args "pretrained=hongweimeng/Qwen3.6-35B-A3B-RTN-INT4-W4A16,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.95,enforce_eager=True" \
  --tasks gsm8k_nothink \
  --include_path . \
  --num_fewshot 5 \
  --batch_size auto

VLLM_USE_TRITON_AWQ=1 is required — the default CUDA AWQ kernel enforces OC % group_size == 0, which some MoE expert layers violate at small batch sizes. The Triton kernel has no such restriction.

Comparison with UINT4

Scheme	Symmetric	Wikitext-2 PPL
uint4_wo_128	No (asymmetric)	6.926
int4_wo_128	Yes	6.990

vLLM inference

Requires vLLM with Quark W4A16 INT4 support. Set VLLM_USE_TRITON_AWQ=1 to use the Triton AWQ kernel, which handles all layer shapes correctly (see GSM8K note above).

import os
os.environ["VLLM_USE_TRITON_AWQ"] = "1"

from vllm import LLM, SamplingParams

llm = LLM(
    model="hongweimeng/Qwen3.6-35B-A3B-RTN-INT4-W4A16",
    quantization="quark",
    dtype="auto",
    gpu_memory_utilization=0.95,
    enforce_eager=True,
)
print(llm.generate(["Hello"], SamplingParams(max_tokens=32))[0].outputs[0].text)

Reproduce

This repo includes scripts/ used to quantize and evaluate:

source scripts/env.sh
bash scripts/01_quantize_rtn_int4.sh   # needs base model + quark + GPU
bash scripts/02_eval_ppl_int4.sh

Base model: Qwen/Qwen3.6-35B-A3B (~67 GB FP16).

Files

• model.safetensors — quantized weights (~21 GB)
• config.json — includes quantization_config with layer excludes
• scripts/ — quantization and PPL evaluation scripts
• eval/ppl_wikitext.json — Wikitext-2 PPL results
• eval/leaderboard/ — Open LLM Leaderboard results (JSON + per-sample JSONL)