Gecko: Versatile Text Embeddings Distilled from Large Language Models
Authors: name: Jinhyuk Lee; name: Zhuyun Dai; name: Xiaoqi Ren; name: Blair Chen; name: Daniel Cer; name: Jeremy R. Cole; name: Kai Hui; name: Michael Boratko; name: Rajvi Kapadia; name: Wen Ding; name: Yi Luan; name: Sai Meher Karthik Duddu; name: Gustavo Hernandez Abrego; name: Weiqiang Shi; name: Nithi Gupta; name: Aditya Kusupati; name: Prateek Jain; name: Siddhartha Reddy Jonnalagadda; name: Ming-Wei Chang; name: Iftekhar Naim
Published: 2024-03-29 17:56:40

Abstract: We present Gecko, a compact and versatile text embedding model. Gecko
achieves strong retrieval performance by leveraging a key idea: distilling
knowledge from large language models (LLMs) into a retriever. Our two-step
distillation process begins with generating diverse, synthetic paired data
using an LLM. Next, we further refine the data quality by retrieving a set of
candidate passages for each query, and relabeling the positive and hard
negative passages using the same LLM. The effectiveness of our approach is
demonstrated by the compactness of the Gecko. On the Massive Text Embedding
Benchmark (MTEB), Gecko with 256 embedding dimensions outperforms all existing
entries with 768 embedding size. Gecko with 768 embedding dimensions achieves
an average score of 66.31, competing with 7x larger models and 5x higher
dimensional embeddings.