bihungba1101/MOSS-Audio-4B-Instruct-FP8-Dynamic

MOSS-Audio-4B-Instruct FP8 Dynamic

Dynamic FP8 quantization of OpenMOSS-Team/MOSS-Audio-4B-Instruct, produced with llm-compressor 0.11.0 and compressed-tensors 0.16.0.

Quantization

• Language-model Linear layers: FP8 weights, per-token dynamic FP8 activations
• Audio encoder, audio adapter, deepstack audio mergers, embeddings, and lm_head: BF16
• Calibration data: none; dynamic activation quantization is data-free
• Hardware used: NVIDIA GeForce RTX 5090

Keeping the audio path in BF16 avoids quantizing the feature extraction and audio-to-language projection components while reducing the language model's weight and activation precision.

vLLM

Verified with vLLM 0.28.0 on an RTX 5090. vLLM selected CutlassFP8ScaledMMLinearKernel for CompressedTensorsW8A8Fp8 and resolved the model as MossAudioModel.

vllm serve bihungba1101/MOSS-Audio-4B-Instruct-FP8-Dynamic \
  --trust-remote-code \
  --max-model-len 2048 \
  --enforce-eager

Audio requests use the OpenAI-compatible chat completions API:

{
  "model": "bihungba1101/MOSS-Audio-4B-Instruct-FP8-Dynamic",
  "messages": [{
    "role": "user",
    "content": [
      {"type": "audio_url", "audio_url": {"url": "https://example.com/audio.wav"}},
      {"type": "text", "text": "Assess pronunciation, fluency, prosody, clarity, pace, rhythm, stress, and intonation."}
    ]
  }]
}

Verification

• Text smoke test: 17 * 6 returned 102.
• Audio inference passed with three real samples from bihungba1101/speech_attempt_raw lasting 31.47, 41.88, and 47.82 seconds.
• All three requests returned non-empty assessments covering pronunciation/accent, fluency, pace, pauses, articulation, and vocal delivery.

The 2048 context limit above is the tested deployment setting, not an architectural maximum. Longer audio consumes more multimodal prompt tokens, so raise --max-model-len if your GPU memory and workload require it.