CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation
  Generation
Authors: name: Yiruo Cheng; name: Kelong Mao; name: Ziliang Zhao; name: Guanting Dong; name: Hongjin Qian; name: Yongkang Wu; name: Tetsuya Sakai; name: Ji-Rong Wen; name: Zhicheng Dou
Published: 2024-10-30 15:06:32

Abstract: Retrieval-Augmented Generation (RAG) has become a powerful paradigm for
enhancing large language models (LLMs) through external knowledge retrieval.
Despite its widespread attention, existing academic research predominantly
focuses on single-turn RAG, leaving a significant gap in addressing the
complexities of multi-turn conversations found in real-world applications. To
bridge this gap, we introduce CORAL, a large-scale benchmark designed to assess
RAG systems in realistic multi-turn conversational settings. CORAL includes
diverse information-seeking conversations automatically derived from Wikipedia
and tackles key challenges such as open-domain coverage, knowledge intensity,
free-form responses, and topic shifts. It supports three core tasks of
conversational RAG: passage retrieval, response generation, and citation
labeling. We propose a unified framework to standardize various conversational
RAG methods and conduct a comprehensive evaluation of these methods on CORAL,
demonstrating substantial opportunities for improving existing approaches.