Qwen3-Embedding-8B-AWQ (W4A16)

AWQ 4-bit (W4A16) quantization of Qwen/Qwen3-Embedding-8B, built with llm-compressor in compressed-tensors format.

What is quantized

Component	Precision
All decoder Linear layers (q/k/v/o_proj, MLP)	INT4 W4A16, g128, symmetric
embed_tokens, RMSNorms	BF16

• 252 Linear layers quantized; on-disk ~4.8 GB (vs ~15 GB BF16).
• Sentence-transformers configs included (last-token pooling + L2 normalize).
• Embedding dim 4096 (MRL: dims 32-4096 supported by the base model).

Quality (AWQ vs BF16)

MTEB-lite on RTX 5060 Ti (sm_120):

Task	BF16	AWQ	Recovery
STSBenchmark Spearman	0.884	0.881	99.6%
SciFact nDCG@10	0.787	0.777	98.8%
STS score Pearson AWQ vs BF16	-	-	0.996

Evals

Full AWQ vs BF16 dump: LostGentoo/awq-quant-evals (qwen3_embedding_8b_awq_vs_bf16.json, plus combined quant_quality_evals.json).

Optimal vLLM serve (embeddings)

Base arch is Qwen3ForCausalLM - convert it to a pooling/embedding runner. compressed-tensors is auto-detected - do not pass --quantization awq.

vllm serve LostGentoo/Qwen3-Embedding-8B-AWQ \
  --runner pooling \
  --convert embed \
  --pooler-config '{"pooling_type":"LAST","task":"embed"}' \
  --trust-remote-code \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --generation-config vllm

Notes:

• --runner pooling --convert embed: required so /v1/embeddings is exposed.
• --pooler-config pooling_type=LAST: matches official Qwen3-Embedding last-token pooling. L2-normalize client-side if the response is not already unit-norm.
• Instructions: for retrieval queries, prefix with Instruct: <task>\nQuery:<text> (documents: raw text, no instruct). See the base model card.
• Matryoshka: for e.g. 1024-d vectors add "dimensions":1024 inside --pooler-config.
• Blackwell: Marlin W4A16 works; add --enforce-eager only if bring-up hits compile issues.

OpenAI-compatible embeddings

curl http://127.0.0.1:8000/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "LostGentoo/Qwen3-Embedding-8B-AWQ",
    "input": [
      "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: What is the capital of France?",
      "Paris is the capital and largest city of France."
    ]
  }'

sentence-transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("LostGentoo/Qwen3-Embedding-8B-AWQ")
emb = model.encode(["The capital of France is Paris."])
print(emb.shape)  # (1, 4096)

transformers (last-token pool)

import torch
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("LostGentoo/Qwen3-Embedding-8B-AWQ", device_map="auto")
tok = AutoTokenizer.from_pretrained("LostGentoo/Qwen3-Embedding-8B-AWQ")
texts = ["The capital of France is Paris."]
inputs = tok(texts, padding=True, return_tensors="pt").to(model.device)
with torch.no_grad():
    hidden = model(**inputs).last_hidden_state
    lengths = inputs["attention_mask"].sum(dim=1) - 1
    emb = hidden[torch.arange(hidden.size(0)), lengths]
    emb = torch.nn.functional.normalize(emb, p=2, dim=1)
print(emb.shape)  # (1, 4096)

Recipe

• llm-compressor 0.12: AWQModifier(duo_scaling=False) + QuantizationModifier(W4A16)
• Calib: 256 samples from WikiText-2-raw, seq 2048
• Load: CPU + sequential onloading (Qwen3DecoderLayer) for 16 GB GPUs

License

Apache-2.0, inherited from the base model.