Jlfn/embeddinggemma2-vq4x4096-q80token

EmbeddingGemma-2 — 4-bit VQ artifact for the ds4 engine

A 4-bit vector-quantized (VQ) build of google/embeddinggemma-2 (revision 914f7f89142e33e77833254d9c9b90c3cef7303b, Apache-2.0), packaged as a GGUF container for the ds4 engine. Text (and one image per item) in, one 768-dimensional L2-normalized vector out.

Where everything lives

What	Where
This artifact's tooling, recipe and gate evidence	https://github.com/JLFN/embeddinggemma2 (public)
The engine that runs it (Rust host + C/CUDA core)	https://github.com/JLFN/embeddinggemma2-ds4-rs (public)
Step-by-step usage guide (every command measured)	https://github.com/JLFN/embeddinggemma2/blob/dev/docs/USAGE.md
The quantization recipe, in full	https://github.com/JLFN/embeddinggemma2/blob/dev/RECIPE.md
Upstream model card	https://huggingface.co/google/embeddinggemma-2

Files

File	Bytes	sha256
EmbeddingGemma-2-914f7f8-vq4x4096-q80token-gemma_vq.gguf	437,508,416	5bcf22161dda46bccbd5123b26d519efccad981cbfe11cce7227415b3373b33f
EmbeddingGemma-2-914f7f8-vision-mmproj.gguf	367,088,960	9ffb86e3eeb5834095e0bd42b6f5fc0505f02b8e208a6037d01db448e6566838

The first file is the text tower (the embedding model). The second is the vision tower (211 tensors) required for the image path; it is loaded only when you pass --vision.

Read this before you try llama.cpp

This is not a llama.cpp-loadable GGUF. It carries the ds4 engine's own family (general.architecture = gemma_vq, family=16 variant=19) and its matrices are the reference quantizer's type-42 vqblob payloads (4-bit, codebook dimension 4096), not Q4_*/IQ* blocks. llama.cpp will refuse the file — by design. Build the engine from the repository above and use ds4-rs.

How to use it

Build the engine (once)

git clone https://github.com/JLFN/embeddinggemma2-ds4-rs
cd embeddinggemma2-ds4-rs
make cuda CUDA_ARCH=sm_89      # or: make cpu

make cuda CUDA_ARCH=<your arch> records the architecture for later builds; use make cuda-generic to let the build detect it.

One-shot text embedding (CLI)

./ds4 -m EmbeddingGemma-2-914f7f8-vq4x4096-q80token-gemma_vq.gguf \
      --embed "The cat sat on the warm windowsill."

JSON on stdout: n_tokens, dims (768), secs, and the embedding array. On an RTX 4070 SUPER this takes about 34 ms. Other sources: --embed-file FILE, --embed-stdin.

Pick the task deliberately — the prefix is prepended before tokenizing, so it changes the vector. Embed queries with a query task and documents with Retrieval-document, then compare across the two:

./ds4 -m <artifact> --embed "your question"      --task SearchQuery
./ds4 -m <artifact> --embed-file page.txt        --task Retrieval-document
./ds4 -m <artifact> --embed "..."                --dims 256   # 512 | 256 | 128, renormalized

The HTTP route

./ds4-server -m EmbeddingGemma-2-914f7f8-vq4x4096-q80token-gemma_vq.gguf \
             --vision EmbeddingGemma-2-914f7f8-vision-mmproj.gguf \
             --model-id eg2vq --host 127.0.0.1 --port 8791 --backend cuda

curl -s localhost:8791/health
curl -s -X POST localhost:8791/v1/embeddings -H 'Content-Type: application/json' \
  -d '{"input":"Vad gäller för övertidsersättning?","task":"SearchQuery","model":"eg2vq"}'

Start the server or the CLI, not both at once: they take the same machine-wide model lock. GET /health reports the contract you can program against (item_tokens_max, batch_rows_max, and the image window).

Many items in one call

The route takes an array and drives one tower call per request, which is where the throughput is:

jq -n --rawfile a doc1.txt --rawfile b doc2.txt \
   '{input:[$a,$b],task:"Retrieval-document",model:"eg2vq"}' \
 | curl -s -X POST localhost:8791/v1/embeddings -H 'Content-Type: application/json' -d @-

Measured: 143 pages of a Swedish collective agreement (76,504 tokens) embedded in 19.0 s, about 4,029 tokens/s.

Images

One image per item, and the item's text must mark its position with <|image|>. The item is the OpenAI content-part shape, with the image as a base64 data URI:

jq -n --rawfile u uri.txt \
   '{input:[{content:[{type:"text",text:"<|image|>"},
                      {type:"image_url",image_url:$u}]}],task:"Raw",model:"eg2vq"}' \
 | curl -s -X POST localhost:8791/v1/embeddings -H 'Content-Type: application/json' -d @-

Build that JSON from a file: a page-sized data URI is far past the shell's argument-list limit.

Turning a PDF into a searchable index

Nothing here reads PDF; convert first. For a PDF with a text layer, extract and batch (this is what produced the 19-second number above):

pdftotext agreement.pdf - \
  | jq -Rs '[split("\f")[] | select((gsub("\\s";"")|length) > 0)]' > pages.json

For a scan with no text layer, render pages and send them as images:

pdftoppm -f 19 -l 19 -r 150 -png agreement.pdf page19

The full worked example, including the query loop that ranks the index, is in docs/USAGE.md.

How it was produced

The weights are the reference implementation's own (YoungAi quantizer, the same toolchain as the DeepSeek V4.1 recipe): kmeans codebook, GPTQ column-group feedback, the payload writer and the block-scalar encoders were called, not reimplemented. The container is this project's DQVD v1 wrapper, re-emitted as GGUF. The artifact's own metadata records the recipe verbatim:

towers=vq4x4096  sharedcb=0 cbfp16 token=q8_0 frontend=vq other=keep nocal
encoder=cpu (eg2_pack) src=914f7f89142e

1376 tensors: 461 vqblob, 1 q8_0 (the 262144×512 token table), 914 bf16 kept at source precision (norms, biases, scales, the vision patch embedder). 42 metadata keys, the gemma4 tokenizer with the checkpoint's own ids.

Measured quality

Against the frozen llama.cpp reference and a standard retrieval benchmark:

Measurement	Result
Tokenizer ids vs llama.cpp over the frozen set	113 items, 7088 tokens, 0 mismatches
Text tower on the baseline weights vs the frozen anchor	mean cosine 0.99999681, min 0.99998727
This 4-bit artifact vs the frozen anchor	mean cosine 0.96007, min 0.82945 — the codec's cost
BEIR SciFact (300 queries / 283 docs / 339 pairs), 768 dims	baseline MRR@10 0.951778 / recall@10 0.983333; this artifact 0.935819 / 0.990000
Device (CUDA) tower vs the CPU tower	min cosine 0.999999950, max rel RMS 3.149e-04 (bound 1e-03)

In plain terms: the 4-bit codec costs about 1.6 points of MRR@10 on BEIR SciFact and gains slightly on recall@10. On a 143-page Swedish collective agreement indexed page-by-page, four out of four test queries returned the correct section at rank 1 (cosines 0.7494–0.8359).

Runtime contract

{"item_tokens_max":2048,"batch_rows_max":8192,
 "image":{"soft_tokens_min":70,"soft_tokens_max":1120,
          "pixels_min":161280,"pixels_max":2580480,
          "embd":512,"per_item_max":1,"serial":true}}

• One item ≤ 2048 tokens; one request ≤ 8192 rows total; one image per item.
• Task prefixes come from the checkpoint's own prompt map (SearchQuery, Retrieval-document, STS, Classification, …; Raw for none).
• --dims 512|256|128 truncates Matryoshka-style and renormalizes.
• The CLI embeds one item per process; batching is the HTTP route's job.
• The audio tower is not part of this artifact.

License and attribution

The base model is Apache-2.0 (Google DeepMind). This is a derivative quantization; the Apache-2.0 license text is included as LICENSE, and the upstream model card and license remain in force for the underlying weights: https://ai.google.dev/gemma/docs/gemma_4_license

Quantization and container: this project's toolchain — see RECIPE.md and docs/ for the byte-level accounting and every gate.