MehulChourasia/sarvam-30b-compressed

Sarvam-30B — Energy-Efficient 4-bit Compression (experts + dense, incl. lm_head)

A compressed build of sarvamai/sarvam-30b for low-energy inference on a single A100, served with vLLM. Every matmul weight matrix is 4-bit — the routed experts and all the dense per-token weights, including the large lm_head — in the compressed-tensors format. Licensed Apache-2.0 (same as the base model).

Serve exactly as the challenge specifies:

vllm serve <this-repo> --config vllm_config.yaml

Results

Measured on one A100. Energy is the GPU's NVML energy over a fixed generative workload under batched inference; accuracy uses an identical harness for the baseline and this model. Recovery = compressed / baseline. These are internal proxy benchmarks (GSM8K / IFEval / MedMCQA / ARC-Challenge), scored generatively via the served chat endpoint; treat them as estimates (n ≈ 100).

	bf16 baseline	This model (4-bit everything)
Energy (GPU, NVML, batched)	100%	~41% of baseline
On-disk size	~64 GB	~17 GB
Math (GSM8K, generative)	0.86	0.87
Writing (IFEval, prompt-strict)	0.23	0.21
Medical (MedMCQA, generative)	0.72	0.67
Questions (ARC-Challenge, generative)	0.88	0.91
Mean recovery	100%	~97%

The energy reduction comes from reading ~4× fewer weight bytes per token. (Math and Questions land slightly above the baseline — that is n≈100 sampling noise; the point is the quantization is near-lossless on average.)

Multilingual (Indic) recovery

sarvam-30b is built for Indian languages, so we also checked that 4-bit quantization does not disproportionately hurt low-resource languages. Global-MMLU (professionally translated MMLU), scored generatively via the chat endpoint, n ≈ 100/language:

Language	bf16 baseline	This model (4-bit)	Recovery
Hindi (hi)	0.84	0.79	94.0%
Bengali (bn)	0.70	0.71	101.4%
Telugu (te)	0.76	0.73	96.1%
Mean	—	—	~97%

Indic recovery (~97%) matches the English-suite average — quantization is near-lossless across both Indo-Aryan (Hindi, Bengali) and Dravidian (Telugu) languages, at the same ~45% energy as the rest of the suite.

1) Compression approach

sarvam-30b is a fine-grained Mixture-of-Experts model (128 routed experts + 1 shared, top-6 routing, 19 layers; ~32B params, ~2.4B active per token). Inference is memory-bandwidth bound, so energy per token is dominated by the weight bytes read from HBM. We therefore quantize all of the heavy matmul weights.

Technique: post-training quantization (GPTQ), compressed-tensors format. No pruning, no distillation.

• Quantized to 4-bit (group size 128, symmetric int):
  • routed experts (model.layers.{1..18}.mlp.experts.{0..127}.{gate,up,down}_proj),
  • attention (query_key_value, dense), the shared expert, the layer-0 dense FFN,
  • and lm_head (vocab 262144 × 4096).
• Kept high-precision: the router (mlp.gate, fp32) — quantizing it scrambles expert selection and collapses quality; input embeddings and all norms (bf16).
• MoE-aware calibration: every expert is calibrated (not only the ones the calibration prompts happen to route to), so rarely-activated experts quantize well.

Why quantizing lm_head matters: it runs on every decoded token (262k-vocab output projection) and is ⅓ of the per-token decode cost — the single biggest energy term after the experts. Taking it from 8-bit/bf16 down to 4-bit is the largest remaining energy lever, and it is near-lossless here (1.5 pt of recovery).

2) vllm_config.yaml arguments

• quantization: compressed-tensors — embedded in config.json and auto-detected; set explicitly for clarity. vLLM selects Marlin WNA16 4-bit kernels automatically.
• max-model-len: 16384 — must hold the prompt plus a long reasoning trace. Math benchmarks can emit ~8k reasoning tokens, so a tight 8192 window would reject those requests (prompt + generation overflows). 16384 is safe headroom; the model is only ~17 GB, so the extra KV reservation barely affects batching or energy (still far below vLLM's 50000 default).
• trust-remote-code: true — sarvam_moe ships a custom config (auto_map).
• enable-prefix-caching: true — reuses KV for shared prompt prefixes; a small, free Wh saving at zero accuracy cost.

3) Notes for running the evaluation

• Engine: built and validated for vLLM (latest). Loads cleanly with vllm serve <repo> --config vllm_config.yaml on a single A100; vLLM serves the 4-bit experts, dense linears, and lm_head with Marlin WNA16 kernels plus FlashAttention — all selected automatically, no custom kernels required.
• Exact environment is pinned in requirements.txt (key: vllm==0.23.0, torch==2.11.0, transformers==5.12.0, compressed-tensors==0.17.0).
• If your environment lacks a CUDA toolkit for FlashInfer's JIT sampler, set VLLM_USE_FLASHINFER_SAMPLER=0 (negligible effect).
• ⚠️ KNOWN ERROR — max_new_tokens must fit the model's context window. This model is served with max-model-len: 16384. If the evaluation sends a max_new_tokens / max_tokens larger than the remaining context (e.g. the default 65536), vLLM rejects every request with HTTP 400 (prompt + max_tokens > max_model_len) and the model returns no output. Fix: reduce max_new_tokens so that prompt + generation stays within 16384 (the model's max-model-len). The model's reasoning fits comfortably in this window (typically ~3k tokens), so capping the generation length does not affect accuracy.

Reproduction

The scripts used are in scripts/:

• quantize.py — GPTQ recipe (experts + all dense incl. lm_head → 4-bit, router fp32, MoE-aware calibration) that produces this model.
• energy_logger.py — NVML GPU-energy (Wh) measurement.
• measure_one.sh / genmc.py — serve + per-category energy (Wh) + generative accuracy.