InteractComp: Evaluating Search Agents With Ambiguous Queries
Authors: name: Mingyi Deng, name: Lijun Huang, name: Yani Fan, name: Jiayi Zhang, name: Fashen Ren, name: Jinyi Bai, name: Fuzhen Yang, name: Dayi Miao, name: Zhaoyang Yu, name: Yifan Wu, name: Yanfei Zhang, name: Fengwei Teng, name: Yingjia Wan, name: Song Hu, name: Yude Li, name: Xin Jin, name: Conghao Hu, name: Haoyu Li, name: Qirui Fu, name: Tai Zhong, name: Xinyu Wang, name: Xiangru Tang, name: Nan Tang, name: Chenglin Wu, name: Yuyu Luo

Language agents have demonstrated remarkable potential in web search and
information retrieval. However, these search agents assume user queries are
complete and unambiguous, an assumption that diverges from reality where users
begin with incomplete queries requiring clarification through interaction. Yet
most agents lack interactive mechanisms during the search process, and existing
benchmarks cannot assess this capability. To address this gap, we introduce
InteractComp, a benchmark designed to evaluate whether search agents can
recognize query ambiguity and actively interact to resolve it during search.
Following the principle of easy to verify, interact to disambiguate, we
construct 210 expert-curated questions across 9 domains through a
target-distractor methodology that creates genuine ambiguity resolvable only
through interaction. Evaluation of 17 models reveals striking failure: the best
model achieves only 13.73% accuracy despite 71.50% with complete context,
exposing systematic overconfidence rather than reasoning deficits. Forced
interaction produces dramatic gains, demonstrating latent capability current
strategies fail to engage. Longitudinal analysis shows interaction capabilities
stagnated over 15 months while search performance improved seven-fold,
revealing a critical blind spot. This stagnation, coupled with the immediate
feedback inherent to search tasks, makes InteractComp a valuable resource for
both evaluating and training interaction capabilities in search agents. The
code is available at https://github.com/FoundationAgents/InteractComp.