ElytronSystems/Nemotron-3.5-Rerank-pulsar-k4

Nemotron 3.5 Rerank 8B: pulsar container, EXL3 4.0 bpw

A ready-to-serve container for pulsar, our inference engine. It holds the text trunk of nvidia/Nemotron-3.5-Rerank-8B-BF16 quantized to EXL3 4.0 bpw, and its score head as NVIDIA ships it. It's 4.5 GB on disk and uses about 5.7 GB of GPU memory.

We run it on a Quadro RTX 5000 (Turing, sm_75) next to the Embed-8B and Parse containers, all three in one pulsar process (12.6 GB of the card's 16 GB).

Parameters: 7.95B in this container, from NVIDIA's 8.38B Rerank-8B; the rest is the upstream vision parts, which are left out. Hugging Face's auto-detected count is lower because it counts EXL3's packed bytes as parameters.

What's inside

Decoder Linears (attention q/k/v/o, MLP gate/up/down)	EXL3 4.0 bpw
Score head, token embedding, norms	as NVIDIA ships them (BF16)
Vision tower, multimodal projector	not included: text only

The EXL3 weights were quantized with exllamav3 1.6.4 (250 × 2048 calibration rows, mul1 codebook) and repacked byte-for-byte into pulsar's container. SHA256SUMS covers every file.

Running it

Build pulsar from the dev branch (any commit from 5139ba62 on) for your GPU, then:

make pulsar-server CUDA_ARCH=sm_75          # Turing; we have only served it on Turing
pulsar-server -m /path/to/this/repo --host 0.0.0.0 --port 8400

POST /v1/rerank takes the Cohere / Jina shape:

curl -s localhost:8400/v1/rerank -H 'Content-Type: application/json' \
  -d '{"query": "how much should a power supply be derated",
       "documents": ["Derate power supplies to 80% of rated load.", "The cafeteria opens at noon."], "top_n": 2}'

Results come best first. relevance_score is the sigmoid of the head's logit; "raw_scores": true returns the logit. The scores run low: relevant passages scored 0.06 to 0.17 in our tests and unrelated ones about 0.02, so don't keep a cut-off tuned for another reranker (a 0.1 threshold drops most good results). Each query-document pair is up to 16,384 tokens.

Quality

On 17 query-document pairs of up to 5,100 tokens:

Compared with	largest score difference	document order
The same EXL3 weights in f32 on the CPU (what the engine should compute)	0.0029	unchanged
The original BF16 model in f32 (the cost of 4-bit weights)	-0.006 .. +0.032	unchanged

On our company documents, reranking the best 30 hybrid-search results with this container raised document MRR from 0.80 to 0.93 (22 questions, a small set).

License

OpenMDW-1.1, as NVIDIA's model (LICENSE). The base model, Ministral-3-8B-Instruct-2512, is Apache-2.0: see NOTICE, which also lists what we changed, and THIRD_PARTY_NOTICES.md. NVIDIA's own model card is kept as NVIDIA_MODEL_CARD.md.