gemma-4-E4B-it-assistant-W4A16

Compressed-tensors pack-quantized W4A16 quantized derivative of google/gemma-4-E4B-it-assistant for speculative decoding / MTP use.

What This Is

• Base model: google/gemma-4-E4B-it-assistant
• Base model relation: quantized derivative
• Architecture: Gemma4AssistantForCausalLM
• Intended use: assistant drafter model for MTP / speculative decoding
• Quantization format: compressed-tensors pack-quantized
• Weight format: symmetric INT4, group-wise, group_size=32
• Non-quantized tensors: preserved as bf16 / int64 passthrough

This checkpoint was produced from the unquantized QAT assistant checkpoint by directly repacking selected bf16 linear weights into compressed-tensors INT4 groups. It is not a bit-identical transcription of a GGUF export.

Quantized Layers

The converted checkpoint includes packed tensors for 22 linear weights:

• pre_projection.weight
• post_projection.weight
• model.layers.{0..3}.self_attn.q_proj.weight
• model.layers.{0..3}.self_attn.o_proj.weight
• model.layers.{0..3}.mlp.gate_proj.weight
• model.layers.{0..3}.mlp.up_proj.weight
• model.layers.{0..3}.mlp.down_proj.weight

Each packed weight has matching *_packed, *_scale, and *_shape tensors.

Validation

Local validation completed with the following results:

• Model load: successful with Transformers + compressed-tensors + accelerate
• Reference tensors: 50
• Converted tensors: 94
• Packed tensors: 22
• Scale tensors: 22
• Shape tensors: 22
• Reference file size: 159,138,208 bytes
• Converted file size: 141,931,784 bytes
• Global reconstruction RMSE: 0.002791
• Global max absolute error: 0.400391
• Worst tensor by max error: post_projection.weight

Benchmark Context

Benchmarks were run on an NVIDIA L40S GPU (AWS) for a WhatsApp-style workload with:

• concurrent users = 1
• no streaming
• 10 benchmark queries

The overall serving conclusion from those tests was:

• Best overall stack: vLLM + QAT (W4A16) + MTP speculative decoding
• Best tested drafter: the original google/gemma-4-e4b-it-assistant
• This converted assistant checkpoint is structurally compatible, but was slower than the original assistant drafter in vLLM

Serving Stack Results

The following benchmark numbers were measured across 10 WhatsApp-style queries.

To control for different total output lengths between runs, the table also includes a latency normalized to 1024 output tokens per message using:

normalized_latency_1024 = avg_latency_per_message * (1024 / average_output_tokens_per_message)

Setup	TTFT	Decode Speed	Total Latency / Msg	Normalized Latency @ 1024 tokens	Effective Speed
SGLang (Full)	766 ms	66.3 tok/s	13.93 s	16.35 s	62.6 tok/s
vLLM (Full)	738 ms	68.3 tok/s	13.66 s	15.85 s	64.6 tok/s
vLLM (QAT Quantized)	727 ms	130.3 tok/s	7.52 s	8.71 s	117.6 tok/s
vLLM (NVFP4 Quantized, prithivMLmods/gemma-4-E4B-it-NVFP4) + Speculative	739 ms	188.4 tok/s	5.89 s	6.23 s	164.5 tok/s
vLLM (QAT + Speculative, original drafter)	756 ms	180.8 tok/s	5.42 s	6.59 s	155.5 tok/s

These measurements support the following practical takeaways:

• Moving from full weights to the QAT W4A16 checkpoint under vLLM substantially improves decode throughput.
• Adding MTP speculative decoding with the original Google assistant drafter produced the best end-to-end latency and best user-perceived throughput.
• An additional NVFP4 benchmark (prithivMLmods/gemma-4-E4B-it-NVFP4) with the same Google assistant drafter reached higher decode throughput and higher effective throughput than the QAT+speculative run.
• After normalizing all tests to 1024 output tokens per message, the NVFP4 speculative setup also had the best normalized latency (6.23 s vs 6.59 s for QAT+speculative with the original drafter).
• SGLang was not the winning stack in this workload.

Original Drafter vs This Converted Drafter

The converted assistant checkpoint in this repo was also benchmarked as the speculative drafter model in vLLM using:

{"method":"mtp","model":"LeadingPoint/gemma-4-E4B-it-assistant-w4a16","num_speculative_tokens":4,"draft_tensor_parallel_size":1}

Comparison against the original Google assistant drafter:

Drafter Model	TTFT	Decode Speed	Total Latency / Msg	Normalized Latency @ 1024 tokens	Effective Speed
google/gemma-4-e4b-it-assistant	756 ms	180.8 tok/s	5.42 s	6.59 s	155.5 tok/s
LeadingPoint/gemma-4-E4B-it-assistant-w4a16	730 ms	95.6 tok/s	9.75 s	11.58 s	88.4 tok/s

Related benchmark reference using a different main model quantization with the same original Google drafter:

Main Model	Drafter Model	TTFT	Decode Speed	Total Latency / Msg	Normalized Latency @ 1024 tokens	Effective Speed
prithivMLmods/gemma-4-E4B-it-NVFP4	google/gemma-4-e4b-it-assistant	739 ms	188.4 tok/s	5.89 s	6.23 s	164.5 tok/s

Interpretation:

• This converted drafter had slightly lower TTFT.
• But steady-state decode throughput was much slower.
• End-to-end WhatsApp reply latency was substantially worse.
• Effective user-perceived throughput was also much worse than the original Google assistant drafter.
• Even after normalizing to a common 1024 output tokens per message, this converted drafter remained much slower than the original Google drafter (11.58 s vs 6.59 s).
• The NVFP4 benchmark indicates that overall serving performance also depends on the main model quantization, not only the drafter model.

So while this checkpoint is loadable and usable as an assistant model artifact, it did not match the runtime performance of the original Google assistant drafter in the measured vLLM MTP benchmark.

Recommendation

If your goal is the fastest production speculative decoding setup from the tested configurations, the benchmark result favors the original Google drafter over this converted drafter. If you also include normalized latency across different output lengths, the best measured setup in this benchmark set was:

{"method":"mtp","model":"google/gemma-4-e4b-it-assistant","num_speculative_tokens":4,"draft_tensor_parallel_size":1}

running with the NVFP4 main model:

{"model":"prithivMLmods/gemma-4-E4B-it-NVFP4","speculative_config":{"method":"mtp","model":"google/gemma-4-e4b-it-assistant","num_speculative_tokens":4,"draft_tensor_parallel_size":1}}

This converted model should be treated as an experimental compressed-tensors assistant checkpoint rather than a proven performance replacement for the original Google drafter.

Runtime Notes

This model is a Gemma assistant drafter model, not a standalone base LM. It is intended to be referenced as the assistant / draft model in speculative decoding configurations.

Example --speculative-config usage in vLLM once this repo is uploaded:

{"method":"mtp","model":"LeadingPoint/gemma-4-E4B-it-assistant-W4A16","num_speculative_tokens":2,"draft_tensor_parallel_size":1}

For parity with the benchmark configuration, the tested speculative decoding setting was:

{"method":"mtp","model":"google/gemma-4-e4b-it-assistant","num_speculative_tokens":4,"draft_tensor_parallel_size":1}

Required Runtime

Validated locally with:

pip install --upgrade transformers compressed-tensors accelerate

Source Files

This repo includes:

• model.safetensors
• config.json
• tokenizer.json
• tokenizer_config.json
• generation_config.json
• chat_template.jinja

License

Gemma usage remains subject to the upstream Gemma license and terms.