anudit/text-embeddings-inference-pr943-sm86

text-embeddings-inference 1.9.4 + PR #943 (Gemma3 mask fix), prebuilt for sm_86

A prebuilt text-embeddings-router binary of text-embeddings-inference built from PR #943, so you can serve google/embeddinggemma-300m correctly without compiling TEI or using Docker.

Why: every official TEI release with Gemma3 support (checked 1.8.3 and 1.9.4) drops EmbeddingGemma's attention mask when a batch has no padding (all inputs the same length, or a single input). Inputs over ~256 tokens then drift (cosine vs. reference: 0.991 at 512 tokens, 0.967 at 1024, 0.949 at 2048). PR #943 fixes it. This build matches sentence-transformers fp32 at cosine ≥ 0.9999997 on long unpadded batches.

Build

Source	PR #943 head 49f2293ed9020c62a8738b4eb76f2ad049568aa5 (upstream main = v1.9.4 + 1 commit)
Features	-F candle-cuda-volta -F static-linking -F http --no-default-features
CUDA_COMPUTE_CAP	86
CUDA toolkit	12.8 (nvcc 12.8.93), Rust 1.92.0, Ubuntu 24.04
Dynamic deps	libcuda.so.1 (host driver), glibc ≥ 2.39, libstdc++6, OpenSSL 3 (libssl3); no CUDA toolkit or cuBLAS needed
sha256 (binary)	cf0195bb741f9880d93f8a64d30b84e7f7b7ccdd099d53472b06482d4203458a

build_tei.sh is the exact build script. tei.sh / tei.conf are the supervisor service files it was run with.

Requirements

• GPU with compute capability 8.6: RTX 3060–3090, A10, A40, A2... Other archs need a rebuild with their CUDA_COMPUTE_CAP.
• NVIDIA driver supporting CUDA ≥ 12.8 (driver ≥ 570). On an older driver it crash-loops with CUDA_ERROR_UNSUPPORTED_PTX_VERSION ... copy2d_f32.
• x86_64 Linux with glibc ≥ 2.39 and OpenSSL 3 (Ubuntu 24.04 or newer; the Vast PyTorch images qualify).

Use

hf download anudit/text-embeddings-inference-pr943-sm86 --local-dir /opt/tei-pr943-dl
cd /opt/tei-pr943-dl && sha256sum -c text-embeddings-router.zst.sha256
mkdir -p /opt/tei-pr943/bin && zstd -d text-embeddings-router.zst -o /opt/tei-pr943/bin/text-embeddings-router
chmod +x /opt/tei-pr943/bin/text-embeddings-router
(cd /opt/tei-pr943/bin && sha256sum -c /opt/tei-pr943-dl/text-embeddings-router.sha256)

hf download google/embeddinggemma-300m --local-dir /workspace/models/embeddinggemma-300m   # gated: accept the license first
/opt/tei-pr943/bin/text-embeddings-router --model-id /workspace/models/embeddinggemma-300m \
  --hostname 127.0.0.1 --port 18080 --dtype float32 --max-batch-tokens 65536 \
  --max-client-batch-size 512 --max-concurrent-requests 8192 --payload-limit 100000000
curl -s localhost:18080/info   # "sha": "49f2293ed9020c62a8738b4eb76f2ad049568aa5"

Settings that matter for EmbeddingGemma: --dtype float32 (TEI refuses Gemma3 in fp16), a local --model-id path, --max-batch-tokens 65536 on 24 GB / 32768 on 10–12 GB, --payload-limit 100000000 for batches of long docs.

Measured on an RTX 3090 (fp32): ~19–20k tok/s on short docs, ~17k tok/s on 512–1000-token docs, ~10k tok/s on 2048-token docs.

License

Apache-2.0, same as text-embeddings-inference (see LICENSE). Unofficial build, not affiliated with Hugging Face.