CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation
  Generation
Authors: name: Yiruo Cheng, name: Kelong Mao, name: Ziliang Zhao, name: Guanting Dong, name: Hongjin Qian, name: Yongkang Wu, name: Tetsuya Sakai, name: Ji-Rong Wen, name: Zhicheng Dou

Retrieval-Augmented Generation (RAG) has become a powerful paradigm for
enhancing large language models (LLMs) through external knowledge retrieval.
Despite its widespread attention, existing academic research predominantly
focuses on single-turn RAG, leaving a significant gap in addressing the
complexities of multi-turn conversations found in real-world applications. To
bridge this gap, we introduce CORAL, a large-scale benchmark designed to assess
RAG systems in realistic multi-turn conversational settings. CORAL includes
diverse information-seeking conversations automatically derived from Wikipedia
and tackles key challenges such as open-domain coverage, knowledge intensity,
free-form responses, and topic shifts. It supports three core tasks of
conversational RAG: passage retrieval, response generation, and citation
labeling. We propose a unified framework to standardize various conversational
RAG methods and conduct a comprehensive evaluation of these methods on CORAL,
demonstrating substantial opportunities for improving existing approaches.