mGTE: Generalized Long-Context Text Representation and Reranking Models
  for Multilingual Text Retrieval
Authors: name: Xin Zhang; name: Yanzhao Zhang; name: Dingkun Long; name: Wen Xie; name: Ziqi Dai; name: Jialong Tang; name: Huan Lin; name: Baosong Yang; name: Pengjun Xie; name: Fei Huang; name: Meishan Zhang; name: Wenjie Li; name: Min Zhang
Published: 2024-07-29 03:12:28

Abstract: We present systematic efforts in building long-context multilingual text
representation model (TRM) and reranker from scratch for text retrieval. We
first introduce a text encoder (base size) enhanced with RoPE and unpadding,
pre-trained in a native 8192-token context (longer than 512 of previous
multilingual encoders). Then we construct a hybrid TRM and a cross-encoder
reranker by contrastive learning. Evaluations show that our text encoder
outperforms the same-sized previous state-of-the-art XLM-R. Meanwhile, our TRM
and reranker match the performance of large-sized state-of-the-art BGE-M3
models and achieve better results on long-context retrieval benchmarks. Further
analysis demonstrate that our proposed models exhibit higher efficiency during
both training and inference. We believe their efficiency and effectiveness
could benefit various researches and industrial applications.