LLJYY/OmniLion-NVFP4-W4A16

OmniLion NVFP4 W4A16

This is the licensed later OmniLion serving derivative. It combines the SEA-LION/Qwen3.6 language and vision model, Qwen3-ASR audio encoder, and trained P21 audio projector in one Hugging Face SafeTensors checkpoint.

The decoder's supported linear weights use NVIDIA ModelOpt's W4A16_NVFP4 format:

• weights: packed NVFP4, group size 16
• activations: BF16
• GB10 execution: vLLM MarlinNvFp4LinearKernel
• calibration: none; ModelOpt's weight-only preset requires no calibration forward pass

The visual tower, Qwen3-ASR tower, P21 projector, MTP payload, LM head, embeddings, and Qwen3.6 fused linear-attention projections remain unquantized.

Supported inputs through one native vLLM process:

• text
• image
• video
• audio, up to 30 seconds per item
• joint video + audio in either content-part order

This is not GGUF. The included omnilion-vllm-plugin wheel is required.

Verified runtime

The release was verified on Linux ARM64 with NVIDIA GB10, Python 3.12, vLLM 0.29.0, PyTorch 2.13.0+cu129, Transformers 5.17.0, and Marlin's W4A16 execution path.

Install the appropriate vLLM/PyTorch build for your platform first, then install the bundled plugin:

python -m pip install --no-deps ./omnilion_vllm_plugin-0.1.0-py3-none-any.whl

Serve

export VLLM_PLUGINS=omnilion
export PYTHONNOUSERSITE=1

vllm serve LLJYY/OmniLion-NVFP4-W4A16 \
  --served-model-name OmniLion \
  --host 127.0.0.1 \
  --port 8002 \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --max-num-seqs 1 \
  --gpu-memory-utilization 0.65 \
  --limit-mm-per-prompt '{"image":1,"video":1,"audio":1}' \
  --media-io-kwargs '{"video":{"num_frames":30}}' \
  --chat-template-content-format string \
  --enforce-eager

The checkpoint metadata makes vLLM select Marlin automatically on the verified GB10 runtime. Pin the repository's immutable commit revision for production deployments.

Use /v1/chat/completions with model: "OmniLion". OpenAI-compatible content parts are image_url, video_url, and input_audio.

Thinking

Normal requests retain Qwen3.6's thinking behavior. Deterministic health checks may disable it per request:

{"chat_template_kwargs":{"enable_thinking":false}}

Verification and limitations

The release passed native vLLM text, image, video, real-audio, both joint audiovisual orders, oversized-audio rejection, normal-thinking, authentication, and LiteLLM-route checks. A deterministic multilingual panel scored 8/8 for both this derivative and BF16.

On the verified GB10 lane, W4A16 reduced reported model memory from 51.73 GiB to 24.82 GiB and produced about 2.1× the BF16 output-token throughput in the bounded benchmark. Cold startup was 34.43% slower because Marlin prepares/re-packs quantized weights.

Quantization can change generated text. The real-audio response was valid but not byte-identical to BF16. The bounded canaries and panel are release smoke evidence, not a comprehensive capability evaluation. Native W4A4 activation quantization is not claimed for this artifact.

Training data, calibration data, and evaluation media are not included. Keep raw vLLM private and put an authenticated gateway such as LiteLLM in front for networked deployment.

See BENCHMARKS.md, PROVENANCE.md, LICENSES.md, and release-manifest.json.