Commerce-Search-EmbeddingGemma-2

🇰🇷 한국어 카드: MODEL_CARD.ko.md

A text embedding model for English product search: a query goes in, products that match it come out first. It is google/embeddinggemma-2 with its text tower fine-tuned on the Amazon Shopping Queries (ESCI) training split. The image and audio towers are byte-identical to the base model.

Use it for text→text product retrieval only. Image→text retrieval got worse after this fine-tune (see Limitations).

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("ThakiCloud/Commerce-Search-EmbeddingGemma-2")
model.max_seq_length = 1024

queries = ["waterproof running shoes wide fit"]
products = [
    "Trail Runner GTX. brand: Acme. color: black. Waterproof membrane, wide toe box",
    "Classic leather loafer. brand: Acme. color: brown",
]
q = model.encode(queries, prompt_name="query", normalize_embeddings=True)
d = model.encode(products, prompt_name="document", normalize_embeddings=True)
print(q @ d.T)   # cosine similarity; higher = better match

• Use prompt_name="query" for queries and prompt_name="document" for products. The prompts ship in the model config (task: search result | query: and title: none | text: ). Omitting them lowers quality.
• Product text in our evaluation is title. brand: X. color: Y. bullet points (first 600 chars).
• Embeddings are 768-dimensional. Queries and products must use the same dimension.

Evaluation

ESCI (US) dev split held out from training: 2,073 queries, 40,397 products, the products of all dev queries pooled into one index. Every row below was measured by us with the same harness: same queries, same product strings, query length 256 tokens, product length 1,024 tokens, bf16, one B200 GPU. The Qwen3-Embedding-8B row comes from our earlier run (2026-10-17) on the same split, product strings, lengths and scoring code.

Model	Params	pool nDCG@10	rerank nDCG@10	Δ pool nDCG@10 vs ours [95% CI]
This model	740M total, ~270M text	0.6256	0.7845	—
google/embeddinggemma-2 (base)	740M	0.6045	0.7695	−2.11pp [−2.80, −1.37]
Qwen/Qwen3-Embedding-8B	8B	0.5716	0.7501	−5.40pp [−6.32, −4.54]
BAAI/bge-m3	568M	0.5233	0.7391	−10.24pp [−11.25, −9.30]
Qwen/Qwen3-Embedding-0.6B	0.6B	0.5096	0.7238	−11.60pp [−12.61, −10.56]
BM25	—	0.4552	0.6915	—

• pool nDCG@10: rank all 40,397 products for each query. rerank nDCG@10: rank only the products judged for that query. Gains: Exact 1.0, Substitute 0.1, Complement 0.01, Irrelevant 0.
• CIs are paired bootstrap over queries (2,000 resamples).
• This is training seed 0. Seeds 1 and 2 of the same recipe scored 0.6239 and 0.6236 pool nDCG@10 on the same dev split in our training-time evaluation; seed 0 is the best of the three.
• The sealed ESCI test split was not used.

Measured inference cost (bf16, B200): about 480 products/s at 1,024 tokens, 2.9 GB peak GPU memory. The checkpoint is 1.49 GB.

Limitations

• English only. Trained and evaluated on ESCI US. Korean and other languages are untested.
• Not for image→text retrieval. On a Shopify product-image benchmark, image→text R@1 dropped from 0.613 (base) to 0.559 (−5.4pp); text→image R@1 stayed level (0.681 → 0.683). If you need images, use the base model.
• Classification prompt degraded slightly. Brand k-NN accuracy with the Classification prompt fell from 0.156 to 0.138 on the same benchmark. That prompt was not part of training.
• Relevance follows ESCI's annotation (Exact/Substitute/Complement/Irrelevant on Amazon products). Other catalogs and other notions of relevance may behave differently.
• The full multimodal checkpoint is shipped so it loads like the base model. Text-only use loads the vision and audio towers too; we did not build a stripped text-only variant.

Training

• Data: Amazon Shopping Queries Dataset (ESCI), US locale, training queries minus our dev queries (18,815 queries). Negatives: 2 random ESCI "Irrelevant" products per query plus in-batch negatives.
• Objective: InfoNCE, temperature 0.05, batch 128, 1 epoch, learning rate 2e-5. Text tower only; vision and audio towers frozen.
• Base: google/embeddinggemma-2, revision 914f7f89142e33e77833254d9c9b90c3cef7303b.

License and data

• Weights: Apache-2.0, the same license as the base model.
• Training data: ESCI (Apache-2.0; Reddy et al., 2022). The dataset repository has an open question (issue #21) about rights in the underlying product text. This repository contains no ESCI text.
• Evaluation side benchmark: Shopify product-catalogue (for the image→text limitation above only; not trained on).

Citation

@article{reddy2022shopping,
  title={Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search},
  author={Reddy, Chandan K. and others},
  journal={arXiv preprint arXiv:2206.06588},
  year={2022}
}