RAVine: Reality-Aligned Evaluation for Agentic Search
Authors: name: Yilong Xu, name: Xiang Long, name: Zhi Zheng, name: Jinhua Gao

Agentic search, as a more autonomous and adaptive paradigm of retrieval
augmentation, is driving the evolution of intelligent search systems. However,
existing evaluation frameworks fail to align well with the goals of agentic
search. First, the complex queries commonly used in current benchmarks often
deviate from realistic user search scenarios. Second, prior approaches tend to
introduce noise when extracting ground truth for end-to-end evaluations,
leading to distorted assessments at a fine-grained level. Third, most current
frameworks focus solely on the quality of final answers, neglecting the
evaluation of the iterative process inherent to agentic search. To address
these limitations, we propose RAVine -- a Reality-Aligned eValuation framework
for agentic LLMs with search. RAVine targets multi-point queries and long-form
answers that better reflect user intents, and introduces an attributable ground
truth construction strategy to enhance the accuracy of fine-grained evaluation.
Moreover, RAVine examines model's interaction with search tools throughout the
iterative process, and accounts for factors of efficiency. We benchmark a
series of models using RAVine and derive several insights, which we hope will
contribute to advancing the development of agentic search systems. The code and
datasets are available at https://github.com/SwordFaith/RAVine.