cyankiwi

/

GLM-5.2-AWQ-INT4

like 10

Follow

cyankiwi 675

yes, thanks for the release!

Tested! Its nice and fast:
Here's the writeup:
https://dnhkng.github.io/posts/gh200-benchmarking-part-3-glm52/

@cpatonn this mod improved your AWQ model for me: https://huggingface.co/dnhkng/GLM-5.2-AWQ-INT4-FP8-MTP-delta

It's benchmarked in the blog.

Tested! Its nice and fast:
Here's the writeup:
https://dnhkng.github.io/posts/gh200-benchmarking-part-3-glm52/

@cpatonn this mod improved your AWQ model for me: https://huggingface.co/dnhkng/GLM-5.2-AWQ-INT4-FP8-MTP-delta

It's benchmarked in the blog.

I tried dnhkng/GLM-5.2-AWQ-INT4-FP8-MTP-delta, but the draft acceptance rate was surprisingly low. Even on very simple cases, the average accepted length was lower than 1. Did I miss something in my setup, or do you also see this behavior in your tests? In your blog, you said it can be 3 to 4.

Here is my script output

Local FP8 MTP extraction plan
  FP8 source:       /path/to/GLM-5.2-FP8
  Extract dir:      /path/to/tmp
  MTP tensors:      1569
  Source shards:    model-00136-of-00141.safetensors, model-00137-of-00141.safetensors, model-00138-of-00141.safetensors
GLM-5.2 AWQ MTP graft plan
  AWQ source:       /path/to/GLM-5.2-AWQ-INT4
  MTP source:       /path/to/tmp
  Output:           /path/to/GLM-5.2-AWQ-INT4-REFINED
  AWQ link mode:    copy
  MTP tensors:      1569
  MTP shard files:  mtp-layer78-00001-of-00003.safetensors, mtp-layer78-00002-of-00003.safetensors, mtp-layer78-00003-of-00003.safetensors
Done.
  Wrote /path/to/GLM-5.2-AWQ-INT4-REFINED
  Layer-78 tensors: 1569
  Extra shard bytes: 10032829896

and the vllm metrics is
Per-position acceptance rate: 0.343, 0.026, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 7.4%

I add the quantization config patch to vllm==0.23.0
@dnhkng which vllm version are you using?

@Jubileus-ryz No idea why that might be the case. Ive used it a bit, and it seems fine. How did patching vLLLM go?

@Jubileus-ryz No idea why that might be the case. Ive used it a bit, and it seems fine. How did patching vLLLM go?

same as https://huggingface.co/dnhkng/GLM-5.2-AWQ-INT4-FP8-MTP-delta/blob/main/patches/vllm-awq-fp8-mtp-quant-config.patch
I'm going to check if the _get_mtp_quant_config function really worked, thanks for your reply.

Edit: My bad — my code changes weren't taking effect. After the fix, it works perfectly. Thanks!
@dnhkng

@Jubileus-ryz What are the before/after token generation speeds?

@Jubileus-ryz What are the before/after token generation speeds?

@dnhkng I didn't test it on an open dataset. In my setup, I used 8×H20 with 30k input tokens and ~1k output tokens at concurrency 1. Before: 51 tokens/s, after: 124 tokens/s. I set the MTP length to 3, and the average draft acceptance rate was 57.5%.