Dense Retrievers Can Fail on Simple Queries: Revealing The Granularity
  Dilemma of Embeddings
Authors: name: Liyan Xu, name: Zhenlin Su, name: Mo Yu, name: Jiangnan Li, name: Fandong Meng, name: Jie Zhou

This work focuses on an observed limitation of text encoders: embeddings may
not be able to recognize fine-grained entities or events within the semantics,
resulting in failed dense retrieval on even simple cases. To examine such
behaviors, we first introduce a new evaluation dataset in Chinese, named
CapRetrieval, whose passages are image captions, and queries are phrases
inquiring entities or events in various forms. Zero-shot evaluation suggests
that encoders may fail on these fine-grained matching, regardless of training
sources or model sizes. Aiming for enhancement, we proceed to finetune encoders
with our proposed data generation strategies, which obtains the best
performance on CapRetrieval. Within this process, we further identify an issue
of granularity dilemma, a challenge for embeddings to express fine-grained
salience while aligning with overall semantics. Our dataset, code and models in
this work are publicly released at https://github.com/lxucs/CapRetrieval.