Instructions to use ctr49/embeddinggemma-300m-Q5_K_M-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ctr49/embeddinggemma-300m-Q5_K_M-GGUF with sentence-transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("ctr49/embeddinggemma-300m-Q5_K_M-GGUF")

sentences = [
    "The weather is lovely today.",
    "It's so sunny outside!",
    "He drove to the stadium."
]
embeddings = model.encode(sentences)

similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use ctr49/embeddinggemma-300m-Q5_K_M-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ctr49/embeddinggemma-300m-Q5_K_M-GGUF:Q5_K_M
# Run inference directly in the terminal:
llama cli -hf ctr49/embeddinggemma-300m-Q5_K_M-GGUF:Q5_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ctr49/embeddinggemma-300m-Q5_K_M-GGUF:Q5_K_M
# Run inference directly in the terminal:
llama cli -hf ctr49/embeddinggemma-300m-Q5_K_M-GGUF:Q5_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ctr49/embeddinggemma-300m-Q5_K_M-GGUF:Q5_K_M
# Run inference directly in the terminal:
./llama-cli -hf ctr49/embeddinggemma-300m-Q5_K_M-GGUF:Q5_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ctr49/embeddinggemma-300m-Q5_K_M-GGUF:Q5_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ctr49/embeddinggemma-300m-Q5_K_M-GGUF:Q5_K_M

Use Docker

docker model run hf.co/ctr49/embeddinggemma-300m-Q5_K_M-GGUF:Q5_K_M

LM Studio
Jan
Ollama
How to use ctr49/embeddinggemma-300m-Q5_K_M-GGUF with Ollama:
```
ollama run hf.co/ctr49/embeddinggemma-300m-Q5_K_M-GGUF:Q5_K_M
```
Unsloth Desktop
Docker Model Runner
How to use ctr49/embeddinggemma-300m-Q5_K_M-GGUF with Docker Model Runner:
```
docker model run hf.co/ctr49/embeddinggemma-300m-Q5_K_M-GGUF:Q5_K_M
```

Lemonade

How to use ctr49/embeddinggemma-300m-Q5_K_M-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ctr49/embeddinggemma-300m-Q5_K_M-GGUF:Q5_K_M

Run and chat with the model

lemonade run user.embeddinggemma-300m-Q5_K_M-GGUF-Q5_K_M

List all available models

lemonade list

Atomic Chat

Access EmbeddingGemma Q5 on Hugging Face

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This model is a quantized derivative of google/embeddinggemma-300m. Access and use are subject to Google's Gemma Terms of Use. Please review and agree to those terms before accessing the model files.

EmbeddingGemma 300M Q5_K_M GGUF

Q5_K_M GGUF quantization of Google's google/embeddinggemma-300m, intended for embedding and RAG workloads with llama.cpp.

This is a modified derivative of the original Google model. The original F32 weights, including the SentenceTransformers dense modules, were converted to F32 GGUF and then quantized to Q5_K_M using llama.cpp.

Model

Upstream: google/embeddinggemma-300m
Upstream revision: 57c266a740f537b4dc058e1b0cda161fd15afa75
Original tensor type: F32
Format: GGUF
Quantization: Q5_K_M
Embedding dimensions: 768
Context length: 2048
Pooling: mean
SentenceTransformers dense modules: included
Published file: embeddinggemma-300M-Q5_K_M.gguf
File size: 249977184 bytes
SHA-256: 29EE6B3D0F1BA3D25397D949AB11252E774EFAC98B993D99104653581384477E

Conversion

The GGUF was produced directly from the original F32 Safetensors using llama.cpp build b11026, commit:

b49650adb31f2e49a0d76113aeb1792134fd8413

The F32 GGUF conversion used:

convert_hf_to_gguf.py <source> --sentence-transformers-dense-modules --outtype f32

The resulting F32 GGUF was then quantized with:

llama-quantize embeddinggemma-300M-F32.gguf embeddinggemma-300M-Q5_K_M.gguf Q5_K_M

llama.cpp reported 24 of 316 tensors using fallback quantization because their dimensions are incompatible with the requested K-quant block type. This is part of the resulting Q5_K_M mixed quantization.

See PROVENANCE.md for exact revisions, hashes, sizes and verification details.

Verification

Both the intermediate F32 GGUF and the published Q5_K_M GGUF were successfully loaded with llama.cpp and produced 768-dimensional embeddings.

As a basic quantization sanity check, the same German test input was embedded with both models. The cosine similarity between the F32 and Q5_K_M embeddings was:

0.9964518009031602

This is a conversion sanity check, not a retrieval-quality benchmark or model evaluation.

Usage with llama.cpp

Start an embedding server:

llama-server -m embeddinggemma-300M-Q5_K_M.gguf --embedding --pooling mean

Embeddings are then available through the llama.cpp OpenAI-compatible /v1/embeddings endpoint.

License and terms

This repository contains a modified and quantized derivative of google/embeddinggemma-300m.

Gemma is provided under and subject to the Gemma Terms of Use and the Gemma Prohibited Use Policy.

This repository does not impose an additional substantive model license.

For redistribution information and the applicable terms, see:

NOTICE
GEMMA_TERMS_OF_USE_2026-04-01.html
GEMMA_PROHIBITED_USE_POLICY_2024-02-21.html

Provenance

Full build and artifact provenance is documented in PROVENANCE.md.

Downloads last month: 16

GGUF

Model size

0.3B params

Architecture

gemma-embedding

Hardware compatibility

5-bit

Model tree for ctr49/embeddinggemma-300m-Q5_K_M-GGUF

Base model

google/embeddinggemma-300m

Quantized

(314)

this model