Agents-A1-TextOnly-AWQ-INT4

This is an optimized, text-only 4-bit INT4 AWQ quantization of InternScience/Agents-A1 (a 35B parameter Mixture-of-Experts agentic model with hybrid Mamba-Attention layers and 2M context length) with pruned vocabulary and group size 128.

This optimized checkpoint was created by pruning the multimodal vision encoder and excluding the grafted Multi-Token Prediction (MTP) weights to drastically reduce the VRAM footprint, and then slicing the vocabulary size from 248,320 down to 152,064 tokens (mapping away unused visual/coordinate tokens).

It is designed to be served natively under vLLM V1 with FP8 KV cache support.

Key Work Done & Optimizations

1. Multimodal Pruning: Removed the visual tower (model.visual.) and skipped the MTP heads (model_mtp.safetensors), saving 2.0 GB of VRAM during execution.
2. Vocabulary Pruning: Sliced vocabulary down to 152,064 (standard Qwen-2.5-Text vocab scale) by mapping away 96,256 unused vision/coordinate tokens. This saved ~370 MiB of weight VRAM per GPU.
3. AWQ Quantization (Group Size 128): Utilizes a group size of 128, which reduces the scale overhead to only ~0.54 GB total (saving over 0.8 GB to 1.9 GB of VRAM per GPU compared to group-size-16 NVFP4 weights!).
4. Prefill Tuning: Tuned chunked prefill settings to --max-num-batched-tokens 2048, freeing significant activation memory during execution.

Hardware & Serving Recommendations

GPU Requirements

• Ideal Dual-GPU Setup: Two RTX 5060 Ti GPUs (16GB VRAM each) for budget deployment, or any Blackwell GPU pair (e.g., RTX 5080 / 5090) for larger KV cache headroom.
• VRAM Budget: The model weights occupy 18.9 GB total (8.93 GB per GPU under TP=2). Operating with FP8 KV cache enables massive contexts to scale comfortably.

vLLM Serving Command

vllm serve Cadododoom/Agents-A1-TextOnly-AWQ-INT4 \
  --tensor-parallel-size 2 \
  --quantization compressed-tensors \
  --moe-backend marlin \
  --attention-backend flashinfer \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --max-num-seqs 32 \
  --max-num-batched-tokens 2096 \
  --mamba-cache-mode align \
  --gpu-memory-utilization 0.95 \
  --trust-remote-code