BordIRlines: A Dataset for Evaluating Cross-lingual Retrieval-Augmented
  Generation
Authors: name: Bryan Li; name: Samar Haider; name: Fiona Luo; name: Adwait Agashe; name: Chris Callison-Burch
Published: 2024-10-02 01:59:07

Abstract: Large language models excel at creative generation but continue to struggle
with the issues of hallucination and bias. While retrieval-augmented generation
(RAG) provides a framework for grounding LLMs' responses in accurate and
up-to-date information, it still raises the question of bias: which sources
should be selected for inclusion in the context? And how should their
importance be weighted? In this paper, we study the challenge of cross-lingual
RAG and present a dataset to investigate the robustness of existing systems at
answering queries about geopolitical disputes, which exist at the intersection
of linguistic, cultural, and political boundaries. Our dataset is sourced from
Wikipedia pages containing information relevant to the given queries and we
investigate the impact of including additional context, as well as the
composition of this context in terms of language and source, on an LLM's
response. Our results show that existing RAG systems continue to be challenged
by cross-lingual use cases and suffer from a lack of consistency when they are
provided with competing information in multiple languages. We present case
studies to illustrate these issues and outline steps for future research to
address these challenges. We make our dataset and code publicly available at
https://github.com/manestay/bordIRlines.