RARE: Retrieval-Aware Robustness Evaluation for Retrieval-Augmented
  Generation Systems
Authors: name: Yixiao Zeng; name: Tianyu Cao; name: Danqing Wang; name: Xinran Zhao; name: Zimeng Qiu; name: Morteza Ziyadi; name: Tongshuang Wu; name: Lei Li
Published: 2025-06-01 02:42:36

Abstract: Retrieval-Augmented Generation (RAG) enhances recency and factuality in
answers. However, existing evaluations rarely test how well these systems cope
with real-world noise, conflicting between internal and external retrieved
contexts, or fast-changing facts. We introduce Retrieval-Aware Robustness
Evaluation (RARE), a unified framework and large-scale benchmark that jointly
stress-tests query and document perturbations over dynamic, time-sensitive
corpora. One of the central features of RARE is a knowledge-graph-driven
synthesis pipeline (RARE-Get) that automatically extracts single and multi-hop
relations from the customized corpus and generates multi-level question sets
without manual intervention. Leveraging this pipeline, we construct a dataset
(RARE-Set) spanning 400 expert-level time-sensitive finance, economics, and
policy documents and 48,322 questions whose distribution evolves as the
underlying sources change. To quantify resilience, we formalize
retrieval-conditioned robustness metrics (RARE-Met) that capture a model's
ability to remain correct or recover when queries, documents, or real-world
retrieval results are systematically altered. Our results show that RAG systems
exhibit surprising vulnerability to perturbations, with document robustness
consistently being the weakest point regardless of generator size or
architecture. RAG systems consistently show lower robustness on multi-hop
queries than single-hop queries across all domains.