Qwen2.5-3B-Instruct-int4-ov

Qwen/Qwen2.5-3B-Instruct converted to OpenVINO IR with INT4 weight-only quantization (NNCF, group_size=128, asymmetric, ratio=1.0). Intended for serving with OpenVINO Model Server (OVMS) on CPU / Intel GPU.

Serve with OVMS

docker run -d --rm -p 8000:8000 -v $(pwd)/models:/models --device /dev/dri \
  --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) \
  openvino/model_server:2026.2.1-gpu \
  --rest_port 8000 --model_repository_path /models \
  --source_model Prog-up/Qwen2.5-3B-Instruct-int4-ov --target_device GPU --task text_generation

Then call the OpenAI-compatible endpoint at POST /v3/chat/completions.

Benchmark

~26.1 tok/s decode throughput (INT4, Intel iGPU, OVMS 2026.2.1, 256-token greedy decode).