ecommerce-product-search-embeddings-base

The quality variant of the ecommerce-product-search-embeddings family: 109M parameters, 768-dim Matryoshka embeddings, fine-tuned from BAAI/bge-base-en-v1.5 on the same 427,655 real Amazon ESCI relevance judgments with the same audited recipe. Pick this over the 33M flagship when recall over a large catalog matters more than encode throughput.

When should I pick this over the small model?

Honest paired comparison on the held-out ESCI test split (8,956 queries, identical protocol):

	small (33M)	this model (109M)	verdict
nDCG@10, judged pools	0.7483	0.7521	paired delta +0.0037, CI [+0.0013, +0.0061] — real but small
Recall@100, 1.22M-product corpus	0.5212	0.5420	+2.1 points, non-overlapping CIs — the real difference
MRR@10, full corpus	0.4220	0.4385	clear win
CPU throughput (queries/s)	2,135	645	3.3× slower — the price

The pattern is exactly what model capacity buys in retrieval: ranking a shortlist barely improves, but finding the right products in a 1.2M-item haystack improves meaningfully. If your catalog is a few thousand products, keep the small model. If it's six figures and recall is your bottleneck — or you batch-encode offline where 645 q/s is still plenty — this is the better tool.

How do I use it (copy-paste)?

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("albertobarnabo/ecommerce-product-search-embeddings-base")
q = model.encode_query(["pan that doesnt stick eggs"])      # prompt baked in
d = model.encode_document(["T-fal Nonstick Fry Pan 12 Inch"])
print(model.similarity(q, d))

Matryoshka: shrink the index without changing models

Trained with MatryoshkaLoss over [768, 384, 128, 64] — truncate + renormalize:

dims	nDCG@10	vs the 33M model
768 (native)	0.7521	—
384	0.7504	≈ small@384 (0.7483) at the same index size
128	0.7446	> small@128 (0.7423)
64	0.7372	> small@64 (0.7327), and above zero-shot bge-small@384 (0.7250) at 1/12th the width

model = SentenceTransformer("albertobarnabo/ecommerce-product-search-embeddings-base", truncate_dim=128)

Training & evaluation

Identical audited pipeline to the flagship — same 115,631 real pairs + labeled synthetic noisy-query augmentation, rotating graded hard negatives (2 Substitute + 1 Irrelevant), GradCache batch 256, cosine schedule to convergence (dev peak at step 1050 of 1800), checkpoint selection on the published graded metric, zero train/test query overlap. Differences: lr 2e-5, Matryoshka dims [768, 384, 128, 64]. Full protocol, leak guards, and limitations: see the flagship card — everything there applies.

Typo-robustness note (same synthetic-noise dev eval as the family): clean 0.7540 → noised 0.7196, a −3.4-point degradation — same pattern as every model in this family, still worse than MiniLM's −1.9; we say so rather than hide it.

Citation

@article{reddy2022shopping,
  title={Shopping Queries Dataset: A Large-Scale {ESCI} Benchmark for Improving Product Search},
  author={Reddy, Chandan K. and M{\`a}rquez, Llu{\'i}s and Valero, Fran and Rao, Nikhil and Zaragoza, Hugo and Bandyopadhyay, Sambaran and Biswas, Arnab and Xing, Anlu and Subbian, Karthik},
  journal={arXiv preprint arXiv:2206.06588},
  year={2022}
}

Family: flagship 33M · reranker · static ~31MB · dataset