GTrush/Qwen3-VL-Reranker-8B-Q8_0-GGUF-llama_cpp

Qwen3-VL-Reranker-8B — Q8_0 GGUF (llama.cpp)

Working Q8_0 GGUF (including mmproj) of Qwen/Qwen3-VL-Reranker-8B for llama.cpp. Converted from llama.cpp master using the official convert_hf_to_gguf.py.

Available files

File	Quant	Size	Description
Qwen3-VL-Reranker-8B-Q8_0.gguf	Q8_0	8.04 GB (7.50 GiB)	Main language & reranker model (cls.output.weight + RANK pooling)
mmproj-Qwen3-VL-Reranker-8B-Q8_0.gguf	Q8_0	752 MB (718 MiB)	Multimodal vision encoder & DeepStack projector (required for image inputs)

Does it work? / Why this conversion?

Yes. Existing community GGUFs such as mradermacher/Qwen3-VL-Reranker-8B-GGUF do not work properly for reranking in llama.cpp because they were exported by generic conversion pipelines as standard generative VLMs — missing the reranker-specific classification tensors, RANK pooling metadata, and the named rerank chat template (see also llama.cpp #16407). Without those, llama-server --reranking either produces broken relevance scores or rejects multimodal /v1/rerank requests.

This repository was converted directly with llama.cpp's official convert_hf_to_gguf.py (which explicitly supports Qwen3-VL-Reranker), producing proper relevance probabilities in [0, 1] for both text-only and multimodal (text + image) queries and documents.

What's different about this GGUF?

The official convert_hf_to_gguf.py detects Qwen3-VL-Reranker and performs reranker-specific steps that generic converters skip:

• Extracts cls.output.weight ([4096, 2], the "yes" / "no" classifier rows) from lm_head
• Sets qwen3vl.pooling_type = RANK metadata
• Sets qwen3vl.classifier.output_labels = ["yes", "no"]
• Bakes in the named rerank chat template required by llama-server for /v1/rerank (including multimodal media marker splicing)
• Names the vision projector with the standard mmproj- prefix (mmproj-Qwen3-VL-Reranker-8B-Q8_0.gguf) so llama-server -hf discovers and loads it automatically

Quick start

Pull and run directly from Hugging Face

llama-server -hf GTrush/Qwen3-VL-Reranker-8B-Q8_0-GGUF-llama_cpp:Q8_0 --reranking --ctx-size 8192 --port 8081

Or with local files:

llama-server \
  -m Qwen3-VL-Reranker-8B-Q8_0.gguf \
  --mmproj mmproj-Qwen3-VL-Reranker-8B-Q8_0.gguf \
  --reranking \
  --pooling rank \
  --ctx-size 8192 \
  --port 8081

Note: Omit --mmproj (or pass --no-mmproj with -hf) if you only need text-only reranking and want to save ~750 MB of VRAM. Use /v1/rerank, not /v1/embeddings.

Text-only reranking

curl http://127.0.0.1:8081/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{
    "query": "Which planet is known as the Red Planet?",
    "documents": [
      "Venus is often called Earth'\''s twin because of its similar size and proximity.",
      "Mars, known for its reddish appearance, is often referred to as the Red Planet."
    ],
    "top_n": 2
  }'

Multimodal (text + image) reranking

curl http://127.0.0.1:8081/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "text": "A city skyline with tall buildings and a river"
    },
    "documents": [
      {
        "text": "A photograph of the Boston skyline at dusk",
        "image": "https://example.com/skyline.jpg"
      },
      "A plain text document can also be mixed in the same request."
    ],
    "top_n": 2
  }'

models.ini example (Router mode)

Using Hugging Face auto-download (mmproj is fetched automatically):

version = 1

[Qwen3-VL-Reranker-8B]
hf-repo     = GTrush/Qwen3-VL-Reranker-8B-Q8_0-GGUF-llama_cpp:Q8_0
reranking   = true
pooling     = rank
ctx-size    = 8192
batch-size  = 2048
ubatch-size = 2048

Or using local file paths:

version = 1

[Qwen3-VL-Reranker-8B]
model       = /path/to/Qwen3-VL-Reranker-8B-Q8_0.gguf
mmproj      = /path/to/mmproj-Qwen3-VL-Reranker-8B-Q8_0.gguf
reranking   = true
pooling     = rank
ctx-size    = 8192
batch-size  = 2048
ubatch-size = 2048

Convert it yourself

git clone --depth 1 https://github.com/ggml-org/llama.cpp.git
pip install -r llama.cpp/requirements/requirements-convert_hf_to_gguf.txt safetensors pyyaml

hf download Qwen/Qwen3-VL-Reranker-8B --local-dir ./Qwen3-VL-Reranker-8B

# Main reranker model (Q8_0)
python llama.cpp/convert_hf_to_gguf.py ./Qwen3-VL-Reranker-8B \
  --outtype q8_0 \
  --outfile Qwen3-VL-Reranker-8B-Q8_0.gguf

# Multimodal vision projector (Q8_0)
python llama.cpp/convert_hf_to_gguf.py ./Qwen3-VL-Reranker-8B \
  --mmproj \
  --outtype q8_0 \
  --outfile mmproj-Qwen3-VL-Reranker-8B-Q8_0.gguf

License

Apache 2.0 — same as the original model.