InteractComp: Evaluating Search Agents With Ambiguous Queries

## Abstract

Abstract Language agents have demonstrated remarkable potential in web search and information retrieval. However, these search agents assume user queries are complete and unambiguous, an assumption that diverges from reality where users begin with incomplete queries requiring clarification through interaction. Yet most agents lack interactive mechanisms during the search process, and existing benchmarks cannot assess this capability. To address this gap, we introduce InteractComp , a benchmark designed to evaluate whether search agents can recognize query ambiguity and actively interact to resolve it during search. Following the principle of easy to verify, interact to disambiguate , we construct 210 expert-curated questions across 9 domains through a target-distractor methodology that creates genuine ambiguity resolvable only through interaction.
Evaluation of 17 models reveals striking failure: the best model achieves only 13.73% accuracy despite 71.50% with complete context, exposing systematic overconfidence rather than reasoning deficits. Forced interaction produces dramatic gains, demonstrating latent capability current strategies fail to engage. Longitudinal analysis shows interaction capabilities stagnated over 15 months while search performance improved seven-fold, revealing a critical blind spot. This stagnation, coupled with the immediate feedback
inherent to search tasks, makes InteractComp a valuable resource for both evaluating and training interaction capabilities in search agents. The code is available at https://github.com/FoundationAgents/InteractComp .

## 1 Introduction

Language agents have demonstrated remarkable potential across diverse domains, including code generation (Zhang et al., 2025; Hong et al., 2024b)
, data analysis (Hong et al., 2024a; Li et al., 2025b; a), information retrieval (Geng et al., 2025; Song et al., 2025), and decision-making (Liu et al., 2025a; Liang et al., 2025).
A notable trend is the rapid development of search agents (OpenAI, 2025d; Google, 2025b), which can handle complex user queries and gather information across the internet by performing search, browse, and reasoning actions (Mialon et al., 2023; Wei et al., 2025).

Figure: Figure 1: Despite rapid progress on complete search queries (BrowseComp: seven-fold over 15 months), agent performance on ambiguous, interaction-dependent queries (InteractComp) has stagnated around 6-14%. This growing disparity reveals a critical blind spot in agent development.
Refer to caption: x1.png

However, these advanced search agents assume user queries are complete and unambiguous. In practice, users begin with incomplete queries admitting multiple plausible interpretations, and only through interaction can the true intent be identified. Yet most search agents lack interactive mechanisms during search. Commercial agents (OpenAI, 2025d) engage in a single clarification, with no further interaction once search begins. When faced with ambiguity, agents confidently commit to assumed queries, leading to incorrect answers and wasted computational resources.

Existing benchmarks cannot assess this capability. Search benchmarks like GAIA (Mialon et al., 2023) and BrowseComp (Wei et al., 2025) provide all necessary resources upfront, enabling agents to proceed without clarifying ambiguous intent. Interaction benchmarks like IN3 (Qian et al., 2024) and Tau-Bench (Yao et al., 2024) focus on general conversation but lack grounding in verifiable search tasks. Neither addresses the question: Can agents recognize query ambiguity and actively interact to gather disambiguating information during search? Without proper assessment of this capability, we cannot
determine whether recent advances in search agents translate to handling real-world scenarios where user intent must be uncovered rather than assumed.

Motivated by this gap, we introduce InteractComp, a benchmark designed to evaluate whether search agents can recognize ambiguity and actively interact to resolve it. Our design follows a core principle: easy to verify, interact to disambiguate. Questions have short, verifiable answers (1-2 words) that are answerable with enough context, yet require interaction to obtain specific details needed for disambiguation. We achieve this through a target-distractor design: questions use only shared attributes of a lesser-known target and a popular alternative, creating genuine ambiguity that search alone cannot resolve. Agents must interact with simulated users to uncover distinctive attributes not given in the initial query. InteractComp contains 210 expert-curated questions across 9 domains in both English and Chinese, validated to ensure interaction is necessary and answers are verifiable.

Systematic evaluation of 17 models confirms our design principle and reveals a striking failure pattern. When provided complete disambiguating context, models achieve strong performance with the best reaching 71.50% accuracy, validating questions are answerable once information is complete. However, even the best model achieves only 13.73% in the full interaction setting, with most models in single digits. This 5× performance gap exposes the core problem: models fail not due to search and reasoning deficits, but systematic overconfidence that prevents them from engaging in interaction despite having access to it.

Scaling experiments confirm this diagnosis. Simply increasing interaction opportunities from 5 to 20 rounds yields minimal improvement (from 14% to 20%), as models barely increase their questioning behavior. In contrast, forcing models to interact before answering produces dramatic gains (from 14% to 40%). Longitudinally, as shown in Figure [1](https://arxiv.org/html/2510.24668v1#S1.F1), interaction capabilities have shown almost no improvement across all models over 15 months, while BrowseComp performance improved seven-fold during the same period. This stagnation is striking given our forced interaction experiments demonstrate the capability is latent rather than absent and readily improvable. This finding, combined with the clean reward signals from search outcomes, makes InteractComp well-suited for RLVR approaches to improve model interaction with humans.

Our contributions are threefold. (1) We introduce InteractComp, a benchmark evaluating interaction capabilities in search scenarios, with clean reward signals enabling future training approaches. (2) We provide diagnostic evidence across 17 models that interaction failure stems from systematic overconfidence rather than capability deficits. (3) We demonstrate through longitudinal analysis that interaction represents a critical blind spot in agent development, with InteractComp providing a foundation for addressing this neglected dimension.

## 2 Related Work

Search Benchmarks and Agents. Recent benchmarks evaluate search agents along two dimensions. Web-scale search benchmarks like BrowseComp (Wei et al., 2025) assess information gathering across the entire web with complete queries, spawning variants for Chinese (Zhou et al., 2025a), multimodal content (Li et al., 2025d), and enhanced questions (Chen et al., 2025). Tool-augmented benchmarks like GAIA (Mialon et al., 2023) and WebWatcher (Geng et al., 2025) additionally require agents to handle multimedia and perform computations. These benchmarks have motivated diverse agent designs. Reinforcement learning approaches like R1-Searcher (Song et al., 2025) and Search-R1 (Jin et al., 2025) learn integrated search-reasoning patterns, while data synthesis methods like WebSailor (Li et al., 2025c) and WebExplorer (Liu et al., 2025b) enhance long-horizon capabilities. Additionally, both manually designed and self-designed search agents (Zhang et al., 2025; Zeng et al., 2025; Teng et al., 2025) have achieved strong performance through careful workflow engineering.

Interaction Benchmarks and Agents. Complementary to search benchmarks, recent work evaluates agents’ interaction capabilities. SWEET-RL (Zhou et al., 2025b) proposes ColBench for multi-turn collaborative reasoning with RL-based credit assignment across turns. UserBench (Qian et al., 2025a) and UserRL (Qian et al., 2025b) create gym environments for training agents on user-centric tasks where goals are underspecified and preferences emerge incrementally. IN3 (Qian et al., 2024) and Tau-Bench (Yao et al., 2024) evaluate implicit intention understanding and tool-agent-user interaction respectively. These benchmarks collectively reveal that current models struggle with proactive clarification and user alignment—for instance, agents uncover fewer than 30% of user preferences through active questioning in UserBench.

However, these benchmarks primarily focus on general conversational settings or tool-use scenarios, lacking grounding in search tasks where intent errors lead to objectively wrong retrieval results. InteractComp differs by evaluating interaction capabilities specifically in search scenarios, where ambiguous queries must be resolved through clarification before effective retrieval can occur, and where search outcomes provide natural reward signals for training interaction strategies.

## 3 The InteractComp Benchmark

**Table 1: A task instance from InteractComp. Tasks in InteractComp comprise an ambiguous query, the simulated user’s context, and a concise answer.**
| Question: Which team-based striking sport features two sides alternating offense and defense, where individuals sequentially hit a high-speed projectile and teammates coordinate to intercept it in the air? Outcomes depend on whether the projectile is intercepted or lands within the valid playing field. Defense relies on wide positioning and collaboration, all offensive players take turns striking, flight speeds often exceed 100 mph, protective gear is required due to impact risk, and the sport is governed by long-standing associations or leagues. | Context: Struck object is a plastic puck, resembling an ice hockey puck. Striking method uses a whip-like swing: the hitter lashes the puck with a long wooden rod. Defenders wield wooden boards, swinging them to block the puck in mid-air. Field is a giant fan shape, about 300 meters long with a 10–12 degree angle. Defensive teams deploy 18–20 players spread across the field to form a defensive line. Scoring is based on distance and landing point: offensive points depend on how far the puck travels and whether it touches the ground. |
| --- | --- |
| Distractor: BaseBall | Answer: Hornussen |

The InteractComp dataset was constructed entirely by human annotators with the assistance of search tools and language models. While BrowseComp (Wei et al., 2025) evaluates complex search and reasoning with complete initial information, InteractComp evaluates whether agents can recognize ambiguity and actively gather necessary context through interaction during the search process. Our core design principle follows “Easy to verify, Interact to disambiguate”: questions have concise answers that are straightforward to verify once found, yet remain ambiguous without interaction to uncover distinguishing details. This section describes the task structure (§[3.1](https://arxiv.org/html/2510.24668v1#S3.SS1)), construction methodology (§[3.2](https://arxiv.org/html/2510.24668v1#S3.SS2)), and dataset statistics (§[3.3](https://arxiv.org/html/2510.24668v1#S3.SS3)).

### 3.1 Task Overview

As shown in Table [1](https://arxiv.org/html/2510.24668v1#S3.T1), each instance comprises an ambiguous question, a context containing distinctive attributes, the correct answer, and a distractor (a popular alternative sharing attributes with the target). The context is hidden from agents but available to a simulated user responder. Agents receive only the ambiguous question and operate with three actions: search to retrieve web information, interact to propose clarifying questions, and answer to provide the final response. The simulated responder replies with ”yes,” ”no,” or ”I don’t know” based solely on context information. Through this process, agents must recognize ambiguity, gather disambiguating details via interaction, and identify the correct answer. Implementation details for both agents and responders are provided in
Appendix [A.1](https://arxiv.org/html/2510.24668v1#A1.SS1) and Appendix [A.2](https://arxiv.org/html/2510.24668v1#A1.SS2).

Figure: Algorithm 1 Data Construction Pipeline

### 3.2 Data construction and Verification

Our construction methodology draws inspiration from BrowseComp’s answer-first approach (Wei et al., 2025), but fundamentally shifts focus from search complexity to ambiguity resolution. The central challenge in constructing such a benchmark is creating questions that appear reasonable yet systematically lack information for confident resolution. We observe that user ambiguity is particularly pronounced when dealing with similar concepts that share overlapping attributes, it is in these scenarios that additional clarification becomes truly necessary rather than merely helpful.

This observation leads us to design a systematic target-distractor methodology. We deliberately pair an target entity with a similar popular entity (the distractor), crafting questions using only their shared attributes while hiding distinctive information as context. This construction ensures that: (1) questions admit multiple plausible interpretations including the popular distractor, making direct answering unreliable; (2) the target answer possesses all described attributes, ensuring verifiability; and (3) distinctive attributes hidden in context provide clear disambiguation paths through interaction. Algorithm [1](https://arxiv.org/html/2510.24668v1#alg1) formalizes this pipeline, which we detail in the following subsections alongside our two-stage verification process.

#### 3.2.1 Construction Process

Annotators receive the following instruction:

“You need to find a pair of entities that are similar but differ in popularity.
Use their shared attributes to construct an ambiguous question, and reserve the remaining distinctive attributes to form the context.”

Following this instruction, the construction proceeds in four steps:
(1) Entity Selection: annotators identify a lesser-known target and a popular distractor sharing overlapping characteristics; (2) Attribute Categorization: attributes are classified as shared (common to both) or distinctive (unique to target); (3) Question Formulation: only shared attributes are used to create questions admitting multiple plausible candidates; (4) Context Formation: distinctive attributes are reserved as context, ensuring question-context pairs uniquely identify the target while questions alone remain ambiguous.

#### 3.2.2 Verification Process

We implement a two-stage verification protocol to ensure data quality and interaction necessity.

Stage 1: Completeness Verification. Independent annotators validate three requirements: (1) the target answer must possess all attributes described in both the question and context, (2) the question-context combination must admit only one valid answer with no plausible alternatives, and (3) instances where annotators identify valid alternative answers are discarded and reconstructed.

Stage 2: Interaction Necessity Validation. We verify that questions truly require interaction through two complementary checks. First, we manually confirm questions cannot be confidently resolved through direct web search, checking the first five Google result pages. Second, we conduct automated testing with three capable models (GPT-5, GPT-5-mini, Claude-Sonnet-4) across 5-round trials where models have access to search but no interaction. Questions successfully answered by two or more models without interaction are flagged as insufficiently ambiguous and undergo revision to strengthen their ambiguity.

### 3.3 Data Statistics

Figure: Figure 2: Topic distribution and question/context length statistics in InteractComp.
Refer to caption: x2.png

In this section, we present statistics on the topic distribution, question and context length distribution of our curated InteractComp dataset.

Topic distribution. Figure [2](https://arxiv.org/html/2510.24668v1#S3.F2) presents the distribution of samples across 9 topic domains in the InteractComp dataset. The most represented categories include Science & Engineering (21.3%), Humanities (18.0%), and Entertainment (16.6%). The dataset also features Business & Economics (11.8%), Law & Politics (8.5%), and Sports (7.1%). Conversely, domains like Medicine & Life Science (5.7%), Academic & Research (4.7%), and General Knowledge & Misc. (6.2%) have fewer samples.

Question and Context Length distribution. Figure [2](https://arxiv.org/html/2510.24668v1#S3.F2) illustrates the distribution of question and context lengths in the InteractComp dataset. Question length predominantly ranges between 40 to 80 words, with the majority falling within this interval. Context length shows a broader distribution, typically spanning from 40 to over 200 words, with peak frequency in the 60-100 word range. These distributions demonstrate that questions are concise yet informative, while contexts provide comprehensive disambiguation information.

Language distribution. The InteractComp dataset comprises bilingual instances with English accounting for 139 samples (66.19%) and Chinese contributing 71 samples (33.81%), enabling evaluation of interaction capabilities across different linguistic contexts.

## 4 Experiments

### 4.1 Experimental Setup

To systematically evaluate agent capabilities across different interaction paradigms, we design a controlled experimental framework that isolates and measures the incremental contribution of core agent capabilities: knowledge recall, information retrieval, and interactive clarification.

Agent Architecture: We employ the ReAct framework (Yao et al., 2023) as our base architecture, implementing three complementary configurations: (1) Answer-only: direct response generation testing pure knowledge recall, (2) Answer+Search: incorporating web search for information retrieval, and (3) Answer+Search+Interact: adding interactive clarification through interact with responder. This design enables measurement of capability increments while maintaining architectural consistency. To further investigate interaction behavior, we implement a forced-interaction variant for ablation studies that requires minimum interaction thresholds before answer generation. Implementation details are provided in Appendix [A.1](https://arxiv.org/html/2510.24668v1#A1.SS1).

Models: We evaluate across diverse model families including proprietary models (GPT-4o-mini, GPT-4o, GPT-4.1, GPT-5, OpenAI o3, Grok-4, Doubao-1.6, Claude-Sonnet-4, Claude-Opus-4, Claude-3.5-Sonnet) and open-weight models (GLM-4.5, Kimi-K2, Deepseek-V3.1, Deepseek-R1, Qwen3-235B-A22B, Qwen2.5). Following established benchmarking practices, we standardize parameters where supported: temperature=0.6, top_p=0.95.We employ GPT-4o (temperature=0.0) as our grader, providing ground truth, agent response, and question context for binary correctness judgments.We implement responder simulation using GPT-4o (temperature=1.0) that provides structured feedback when agents employ the interact action.

Metrics: We evaluate agents across five key dimensions: (1) Interaction Metrics: Round (average number of conversation turns) and percentage of rounds where interact actions are used (IR) measuring behavioral patterns and action utilization; (2) Performance Metrics: Accuracy (Acc.) measuring the percentage of correctly answered queries, and Calibration Error (C.E.) measuring confidence calibration using 5 confidence bins; and (3) Cost: measured in USD reflecting computational resources usage for practical deployment considerations.

**Table 2: Performance comparison of 17 large language models on the InteractComp dataset. The table reports both interaction behaviors like average number of conversation turns(Round) and percentage of rounds where interact actions (IR) are used; final performance like accuracy (Acc. with std in parentheses) and calibration error (C.E.), along with the estimated total cost. Models are grouped into open-weight and closed-weight categories for clarity. Best accuracy is highlighted in bold.**
| Model | Interaction | Performance | Cost($) |  |  |
| --- | --- | --- | --- | --- | --- |
| Round | IR | Acc. | C.E. |  |  |
| Open Weights Models |  |  |  |  |  |
| GLM-4.5 (Zhipu AI, 2025) | 6.91 | 0.25 | 7.14 (±0.48) | 80.64 | 2.16 |
| Kimi-K2 (Moonshot AI, 2025) | 4.95 | 5.98 | 6.51 (±1.53) | 87.10 | 0.75 |
| Deepseek-V3.1 (DeepSeek, 2025a) | 7.26 | 11.60 | 11.74 (±2.71) | 74.79 | 8.84 |
| Deepseek-R1 (DeepSeek, 2025b) | 6.58 | 44.72 | 13.08 (±0.29) | 77.00 | 60.43 |
| Qwen2.5-72B-Instruct (Yang et al., 2024) | 7.45 | 31.88 | 8.08 (±0.73) | 77.57 | 0.15 |
| Qwen3-235B-A22B (Qwen Team, 2025) | 5.64 | 27.75 | 8.89 (±0.72) | 82.63 | 7.47 |
| Proprietary Models |  |  |  |  |  |
| GPT-4o-mini (OpenAI, 2024b) | 4.16 | 73.95 | 7.13 (±0.42) | 37.44 | 0.35 |
| GPT-4o (OpenAI, 2024a) | 5.65 | 9.26 | 7.62 (±0.79) | 79.50 | 8.65 |
| GPT-4.1 (OpenAI, 2025a) | 5.49 | 34.02 | 10.79 (±1.22) | 82.11 | 5.58 |
| OpenAI o3 (OpenAI, 2025c) | 2.96 | 15.03 | 10.00 (±1.44) | 41.96 | 5.04 |
| GPT-5 (OpenAI, 2025b) | 4.33 | 30.87 | 13.73 (±2.55) | 68.67 | 16.85 |
| Grok-4 (xAI, 2025) | 4.92 | 4.55 | 8.40 (±1.24) | 69.00 | 77.55 |
| Gemini-2.5-Pro (Google, 2025a) | 4.65 | 11.09 | 10.28 (±0.37) | 86.52 | 15.04 |
| Doubao-1.6 (ByteDance, 2025) | 3.08 | 10.60 | 6.73 (±0.97) | 84.35 | 1.40 |
| Claude-3.5-Sonnet (Anthropic, 2024) | 5.63 | 27.57 | 8.10 (±1.91) | 80.04 | 13.09 |
| Claude-Sonnet-4 (Anthropic, 2025b) | 6.90 | 10.76 | 7.46 (±1.37) | 79.62 | 19.47 |
| Claude-Opus-4 (Anthropic, 2025a) | 8.55 | 10.86 | 8.10 (±0.96) | 78.42 | 115.47 |

### 4.2 Main Results

Table [2](https://arxiv.org/html/2510.24668v1#S4.T2) presents comprehensive results across 17 models, revealing striking patterns in how different architectures handle ambiguous queries. The results expose fundamental limitations even in state-of-the-art systems, with the highest-performing model (GPT-5) achieving only 13.73% accuracy, demonstrating the benchmark’s challenging nature.

Diverse Interaction Patterns Across Models. Models exhibit dramatically different interaction strategies, creating distinct behavioral profiles. GPT-4o-mini stands out as an extreme case: it asks questions in 73.95% of available rounds, by far the highest interaction rate, yet achieves only 7.14% accuracy—close to GLM-4.5 which barely interacts (0.25% IR). This suggests that excessive questioning without clear purpose can be counterproductive. Conversely, DeepSeek-R1 demonstrates more balanced behavior with 44.72% IR yielding 13.08% accuracy, the highest among open-weight models, indicating that willingness to interact can translate to better performance when used effectively.

Calibration Quality Correlates with Interaction Patterns. A remarkable finding is that models with higher interaction rates often exhibit superior calibration. GPT-4o-mini’s aggressive questioning strategy, while not improving accuracy, results in dramatically better calibration (37.44 CE) compared to low-interaction models like Doubao-1.6 (84.35 CE). This pattern suggests that interaction, even when not optimally targeted, helps models develop more realistic confidence assessments about their knowledge limitations.

Open-Weight vs. Proprietary Model Divide. The performance gap between open-weight and proprietary models is stark and consistent. All open-weight models struggle with interaction rates below 45%, with most falling under 32%. GLM-4.5, Kimi-K2, and Qwen3-235B-A22B show particularly conservative interaction behavior (0.25%, 5.98%, and 27.75% respectively), suggesting that open-weight models may have been trained to minimize uncertain responses rather than seek clarification. In contrast, proprietary models like GPT-4.1 and GPT-5 show more balanced interaction patterns (34.02% and 30.87%), though even they fall short of optimal information-gathering behavior.

These findings collectively demonstrate that current language models, regardless of scale or sophistication, struggle fundamentally with effective information gathering, often exhibiting either excessive conservatism or ineffective over-questioning when faced with genuine ambiguity.

### 4.3 Ablation Analysis

To validate that our benchmark specifically tests interaction abilities rather than general reasoning, we conduct ablation studies across three evaluation modes using 8 representative models.

Table [3](https://arxiv.org/html/2510.24668v1#S4.T3) reveals dramatic performance gaps confirming interaction as the critical missing component. Three key findings emerge: (1) Answer-only mode exposes fundamental limitations, OpenAI o3 achieves only 5.18%, GPT-5 reaches 7.62%, with catastrophic overconfidence (60.94-93.17% calibration errors). (2) Search augmentation provides minimal benefits, o3 increases to just 8.81% and GPT-5 to 9.52%, demonstrating that information retrieval alone cannot resolve ambiguity. (3) Complete contextual information reveals the performance ceiling, o3 soars to 71.50% (13.8× increase), GPT-5 reaches 67.88%, and calibration errors plummet to 7.44%, confirming underlying reasoning capabilities exist but are inaccessible without proper context.

The massive gap between search-only (6.74-9.52%) and with-context (40.93-71.50%) performance validates our benchmark design: interaction to acquire disambiguating information is the true bottleneck, not reasoning ability. Models possess the knowledge to answer correctly but fail at recognizing when and how to seek necessary clarification.

**Table 3: Ablation study comparing model performance under three evaluation settings: answer-only (models respond without additional evidence), search-only (responses based solely on retrieved information), and with-context (responses supported by complete disambiguating context). Results are reported in terms of accuracy (Acc.) and calibration error (C.E.). The best scores in each column are highlighted in bold.**
| Model | answer-only | search-only | with-context |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Acc. | C.E. | Acc. | C.E. | Acc. | C.E. |  |
| GPT-4o | 2.38 | 88.76 | 7.77 | 80.52 | 40.93 | 47.33 |
| GPT-5 | 7.62 | 76.26 | 9.52 | 79.14 | 67.88 | 21.36 |
| OpenAI o3 | 5.18 | 60.94 | 8.81 | 52.62 | 71.50 | 7.44 |
| GLM-4.5 | 2.38 | 84.40 | 6.74 | 82.41 | 64.77 | 22.37 |
| Kimi-K2 | 1.43 | 90.36 | 7.53 | 86.87 | 53.37 | 40.62 |
| Gemini-2.5-Pro | 2.38 | 93.17 | 7.25 | 90.65 | 69.95 | 28.60 |
| DeepSeek-V3.1 | 3.11 | 85.60 | 8.29 | 79.24 | 65.28 | 24.17 |
| Claude-Sonnet-4 | 2.85 | 87.12 | 7.25 | 81.70 | 59.07 | 26.31 |

**Table 4: Scaling analysis of model performance across different interaction rounds (5, 10, and 20) on a 50-question subsample. We report the average number of interact rounds (IRound), accuracy (Acc.), and calibration error (C.E.) for four representative models: GPT-4o-mini, GPT-5, Claude-Sonnet-4, and Deepseek-V3.1.**
| Rounds | GPT-4o-mini | GPT-5 | Claude-Sonnet-4 | Deepseek-V3.1 |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| IRound | Acc. | C.E. | IRound | Acc. | C.E. | IRound | Acc. | C.E. | IRound | Acc. | C.E. |  |
| 5 | 2.00 | 4.00 | 49.50 | 1.14 | 14.00 | 71.50 | 0.16 | 6.00 | 79.90 | 0.38 | 10.00 | 77.00 |
| 10 | 3.62 | 8.00 | 47.60 | 1.76 | 16.00 | 71.54 | 0.70 | 4.00 | 80.24 | 0.74 | 8.00 | 80.30 |
| 20 | 2.76 | 8.00 | 33.20 | 1.90 | 20.00 | 70.06 | 0.78 | 8.00 | 81.84 | 1.54 | 10.00 | 75.20 |

### 4.4 Scaling Analysis

The ablation studies revealed that models possess the capabilities to handle ambiguous queries when given complete context, but fail to gather necessary information through interaction. We investigate whether providing more interaction opportunities (5, 10, and 20 rounds) encourages information gathering. Figure [3](https://arxiv.org/html/2510.24668v1#S4.F3)(a) and Table [4](https://arxiv.org/html/2510.24668v1#S4.T4) present the results.

Results show that models fail to scale interaction usage with available opportunities. Despite quadrupling round limits from 5 to 20, GPT-5 increases interactions from just 1.14 to 1.90, while Claude-Sonnet-4 barely reaches 0.78 interactions per instance. However, models that do interact more achieve better performance—GPT-5 improves from 14.00% to 20.00% accuracy as interactions increase. This reveals systematic overconfidence as the primary bottleneck: models prematurely conclude they have sufficient information despite evidence that continued questioning improves performance.

### 4.5 Forced Interaction Analysis

Figure: Figure 3: Model performance under different rounds constraints.
Refer to caption: x3.png

To test whether interaction underutilization stems from voluntary choice rather than capability deficits, we implement forced interaction protocols that require agents to ask a minimum number of clarifying questions (ranging from 2 to 10) before providing answers, as shown in Figure [3](https://arxiv.org/html/2510.24668v1#S4.F3)(b).

Results reveal dramatic model-specific differences. GPT-5 doubles its accuracy from 20% to 40% when compelled to ask 8 questions, confirming strong reasoning capabilities hindered by voluntary underuse of interaction. However, not all models benefit—Claude-Sonnet-4 shows modest gains while GPT-4o-mini’s performance actually degrades under forced interaction. This demonstrates that effective information acquisition is a distinct capability varying significantly across architectures, suggesting limitations extend beyond overconfidence to fundamental differences in information-seeking strategies.

### 4.6 Longitudinal Study

Tracking 15 months of model development reveals a concerning divergence: while BrowseComp performance improved seven-fold (10% to 70%), InteractComp performance remained stagnant. Recent models like GPT-5, DeepSeek-R1, and GPT-4.1 cluster around 6-14% accuracy with minimal variation over time. This exposes a fundamental blind spot in AI development: rapid progress on search-focused tasks has not translated to progress in interaction-based problem solving. Without explicit focus on interaction capabilities, models advance in reasoning and retrieval while remaining primitive at recognizing ambiguity and gathering clarification—a critical limitation for practical deployment. Figure [1](https://arxiv.org/html/2510.24668v1#S1.F1) illustrates this stark contrast, showing BrowseComp’s steep upward trajectory alongside InteractComp’s flat performance across all evaluated models.

## 5 Conclusion

This paper presents InteractComp, a benchmark designed to evaluate a critical yet overlooked capability of search agents: recognizing and resolving ambiguous queries through active interaction. While existing search benchmarks have driven remarkable progress in retrieval and reasoning, they uniformly assume users provide complete queries from the outset—an assumption that diverges from real-world behavior where users begin with incomplete information needs. By constructing questions that are easy to verify once sufficient context is gathered yet impossible to disambiguate without interaction, InteractComp systematically evaluates whether agents can recognize ambiguity and actively seek clarification during search.

Our evaluation of 17 models reveals systematic overconfidence as the primary bottleneck rather than capability deficits. Models achieve 68-72% accuracy when provided complete context but only 13.73% with interaction available, severely underutilizing clarifying questions despite their access. Forced interaction experiments confirm this is a strategic failure—when compelled to interact, accuracy doubles, demonstrating latent capabilities current strategies fail to engage. Longitudinal analysis reinforces this diagnosis: while BrowseComp performance improved seven-fold over 15 months, InteractComp scores remained stagnant, exposing a critical blind spot where progress in retrieval has not translated to progress in interaction. Beyond diagnosis, the grounded nature of search provides clean reward signals for training, making InteractComp well-suited for reinforcement learning approaches to develop uncertainty-aware, actively interactive agents. We hope this benchmark provides the foundation for systematic progress on this neglected but essential dimension of agent development.

## References

- Anthropic (2024)
Anthropic.
Claude 3.5 sonnet.
[https://www.anthropic.com/news/claude-3-5-sonnet](https://www.anthropic.com/news/claude-3-5-sonnet), 2024.
- Anthropic (2025a)
Anthropic.
Claude opus 4.
[https://www.anthropic.com/claude/opus](https://www.anthropic.com/claude/opus), 2025a.
- Anthropic (2025b)
Anthropic.
Claude sonnet 4.
[https://www.anthropic.com/claude/sonnet](https://www.anthropic.com/claude/sonnet), 2025b.
- ByteDance (2025)
ByteDance.
Doubao 1.6.
[https://seed.bytedance.com/en/seed1_6](https://seed.bytedance.com/en/seed1_6), 2025.
- Chen et al. (2025)
Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, et al.
Browsecomp-plus: A more fair and transparent evaluation benchmark of deep-research agent.
*arXiv preprint arXiv:2508.06600*, 2025.
- DeepSeek (2025a)
DeepSeek.
Deepseek-chat.
[https://huggingface.co/deepseek-ai/DeepSeek-V3.1](https://huggingface.co/deepseek-ai/DeepSeek-V3.1), 2025a.
- DeepSeek (2025b)
DeepSeek.
DeepSeek-R1.
[https://huggingface.co/deepseek-ai/DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1), 2025b.
- Geng et al. (2025)
Xinyu Geng, Peng Xia, Zhen Zhang, Xinyu Wang, Qiuchen Wang, Ruixue Ding, Chenxi Wang, Jialong Wu, Yida Zhao, Kuan Li, et al.
Webwatcher: Breaking new frontiers of vision-language deep research agent.
*arXiv preprint arXiv:2508.05748*, 2025.
- Google (2025a)
Google.
Gemini 2.5: Our most intelligent ai model.
[https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/](https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/), 2025a.
- Google (2025b)
Google.
Gemini deep research.
[https://gemini.google/overview/deep-research/](https://gemini.google/overview/deep-research/), 2025b.
- Hong et al. (2024a)
Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, et al.
Data interpreter: An llm agent for data science.
*arXiv preprint arXiv:2402.18679*, 2024a.
- Hong et al. (2024b)
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber.
MetaGPT: Meta programming for a multi-agent collaborative framework.
In *The Twelfth International Conference on Learning Representations*, 2024b.
URL [https://openreview.net/forum?id=VtmBAGCN7o](https://openreview.net/forum?id=VtmBAGCN7o).
- Jin et al. (2025)
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han.
Search-r1: Training llms to reason and leverage search engines with reinforcement learning.
*arXiv preprint arXiv:2503.09516*, 2025.
- Li et al. (2025a)
Boyan Li, Chong Chen, Zhujun Xue, Yinan Mei, and Yuyu Luo.
Deepeye-sql: A software-engineering-inspired text-to-sql framework.
*arXiv preprint arXiv:2510.17586*, 2025a.
- Li et al. (2025b)
Boyan Li, Jiayi Zhang, Ju Fan, Yanwei Xu, Chong Chen, Nan Tang, and Yuyu Luo.
Alpha-sql: Zero-shot text-to-sql using monte carlo tree search.
*arXiv preprint arXiv:2502.17248*, 2025b.
- Li et al. (2025c)
Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, et al.
Websailor: Navigating super-human reasoning for web agent.
*arXiv preprint arXiv:2507.02592*, 2025c.
- Li et al. (2025d)
Shilong Li, Xingyuan Bu, Wenjie Wang, Jiaheng Liu, Jun Dong, Haoyang He, Hao Lu, Haozhe Zhang, Chenchen Jing, Zhen Li, et al.
Mm-browsecomp: A comprehensive benchmark for multimodal browsing agents.
*arXiv preprint arXiv:2508.13186*, 2025d.
- Liang et al. (2025)
Xinbin Liang, Jinyu Xiang, Zhaoyang Yu, Jiayi Zhang, Sirui Hong, Sheng Fan, and Xiao Tang.
Openmanus: An open-source framework for building general ai agents, 2025.
- Liu et al. (2025a)
Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, et al.
Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems.
*arXiv preprint arXiv:2504.01990*, 2025a.
- Liu et al. (2025b)
Junteng Liu, Yunji Li, Chi Zhang, Jingyang Li, Aili Chen, Ke Ji, Weiyu Cheng, Zijia Wu, Chengyu Du, Qidi Xu, et al.
Webexplorer: Explore and evolve for training long-horizon web agents.
*arXiv preprint arXiv:2509.06501*, 2025b.
- Mialon et al. (2023)
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom.
Gaia: a benchmark for general ai assistants.
In *The Twelfth International Conference on Learning Representations*, 2023.
- Moonshot AI (2025)
Moonshot AI.
Kimi K2.
[https://moonshotai.github.io/Kimi-K2/](https://moonshotai.github.io/Kimi-K2/), 2025.
- OpenAI (2024a)
OpenAI.
Hello GPT-4o.
[https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/), 2024a.
- OpenAI (2024b)
OpenAI.
GPT-4o mini: Advancing cost-efficient intelligence.
[https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/), 2024b.
- OpenAI (2025a)
OpenAI.
Introducing GPT-4.1.
[https://openai.com/index/gpt-4-1/](https://openai.com/index/gpt-4-1/), 2025a.
- OpenAI (2025b)
OpenAI.
Introducing GPT-5.
[https://openai.com/index/introducing-gpt-5/](https://openai.com/index/introducing-gpt-5/), 2025b.
- OpenAI (2025c)
OpenAI.
Introducing openai o3 and o4-mini.
[https://openai.com/index/introducing-o3-and-o4-mini/](https://openai.com/index/introducing-o3-and-o4-mini/), 2025c.
- OpenAI (2025d)
OpenAI.
Introducing deep research.
[https://openai.com/index/introducing-deep-research/](https://openai.com/index/introducing-deep-research/), 2025d.
- Qian et al. (2024)
Cheng Qian, Bingxiang He, Zhong Zhuang, Jia Deng, Yujia Qin, Xin Cong, Zhong Zhang, Jie Zhou, Yankai Lin, Zhiyuan Liu, et al.
Tell me more! towards implicit user intention understanding of language model driven agents.
*arXiv preprint arXiv:2402.09205*, 2024.
- Qian et al. (2025a)
Cheng Qian, Zuxin Liu, Akshara Prabhakar, Zhiwei Liu, Jianguo Zhang, Haolin Chen, Heng Ji, Weiran Yao, Shelby Heinecke, Silvio Savarese, et al.
Userbench: An interactive gym environment for user-centric agents.
*arXiv preprint arXiv:2507.22034*, 2025a.
- Qian et al. (2025b)
Cheng Qian, Zuxin Liu, Akshara Prabhakar, Jielin Qiu, Zhiwei Liu, Haolin Chen, Shirley Kokane, Heng Ji, Weiran Yao, Shelby Heinecke, et al.
Userrl: Training interactive user-centric agent via reinforcement learning.
*arXiv preprint arXiv:2509.19736*, 2025b.
- Qwen Team (2025)
Qwen Team.
Qwen3-235B-A22B.
[https://arxiv.org/abs/2505.09388](https://arxiv.org/abs/2505.09388), 2025.
- Song et al. (2025)
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen.
R1-searcher: Incentivizing the search capability in llms via reinforcement learning.
*arXiv preprint arXiv:2503.05592*, 2025.
- Teng et al. (2025)
Fengwei Teng, Zhaoyang Yu, Quan Shi, Jiayi Zhang, Chenglin Wu, and Yuyu Luo.
Atom of thoughts for markov llm test-time scaling.
*arXiv preprint arXiv:2502.12018*, 2025.
- Wei et al. (2025)
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese.
Browsecomp: A simple yet challenging benchmark for browsing agents.
*arXiv preprint arXiv:2504.12516*, 2025.
- xAI (2025)
xAI.
Grok 4.
[https://x.ai/news/grok-4](https://x.ai/news/grok-4), 2025.
- Yang et al. (2024)
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al.
Qwen2.5 technical report.
*arXiv preprint arXiv:2412.15115*, 2024.
- Yao et al. (2023)
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models.
In *International Conference on Learning Representations (ICLR)*, 2023.
- Yao et al. (2024)
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.
$\tau$-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024.
URL [https://arxiv.org/abs/2406.12045](https://arxiv.org/abs/2406.12045).
- Zeng et al. (2025)
Weihao Zeng, Keqing He, Chuqiao Kuang, Xiaoguang Li, and Junxian He.
Pushing test-time scaling limits of deep search with asymmetric verification.
*arXiv preprint arXiv:2510.06135*, 2025.
- Zhang et al. (2025)
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xiong-Hui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu.
AFlow: Automating agentic workflow generation.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=z5uVAKwmjf](https://openreview.net/forum?id=z5uVAKwmjf).
- Zhipu AI (2025)
Zhipu AI.
Glm-4.5.
[https://z.ai/blog/glm-4.5](https://z.ai/blog/glm-4.5), 2025.
- Zhou et al. (2025a)
Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, et al.
Browsecomp-zh: Benchmarking web browsing ability of large language models in chinese.
*arXiv preprint arXiv:2504.19314*, 2025a.
- Zhou et al. (2025b)
Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li.
Sweet-rl: Training multi-turn llm agents on collaborative reasoning tasks.
*arXiv preprint arXiv:2503.15478*, 2025b.

## Appendix A Appendix

### A.1 Agent Implementation

Our agent implementation is built upon the ReAct framework (Yao et al., 2023), which combines reasoning and acting in a unified architecture. We implement three distinct agent configurations to systematically evaluate different capability combinations:

Configuration 1: Answer-only: The agent directly generates responses using its internal knowledge without external information gathering. This configuration serves as a baseline to measure pure knowledge recall capabilities on ambiguous queries.

Configuration 2: Answer+Search: The agent can perform web search actions to retrieve external information before generating answers. Available actions include:

- •
search(query): Performs web search with the specified query
- •
answer(response, confidence): Provides final answer with confidence score

Configuration 3: Answer+Search+Ask: The full interaction-enabled agent that can additionally request clarification from users. This configuration adds:

- •
ask(question): Poses yes/no questions to gather missing information

Action Space Design - Each agent operates with a maximum of 10 rounds, where each round allows exactly one action. The agent maintains an internal memory of previous actions and observations. For forced interaction experiments, we implement a constraint requiring minimum interaction thresholds before answer generation is permitted.

The complete system prompts and interaction protocols are detailed below.

### A.2 Responder Simulation

We implement a controlled responder simulation using GPT-4o (temperature=1.0) that provides structured feedback when agents employ the ask action. Upon receiving agent queries, the responder evaluates questions against available context and responds with one of three standardized options: ”yes”, ”no”, or ”I don’t know”. The responder state $s_{r}$ consists of the given context and interaction history, with transitions $T_{r}:(s_{r},q_{agent})\rightarrow o_{r}\in\{\text{yes},\text{no},\text{unknown}\}$ conditioned on context-question alignment. While maintaining response diversity through LLM generation, the constrained output format ensures evaluation consistency.

The complete responder prompts are detailed below.

### A.3 Evaluation Protocol

We validate simulation reliability through repeated sampling across identical context–question pairs across $k=3$ trials,
indicating stable behavior despite the stochastic generation process.
We employ GPT-4o (temperature $=0.0$) as our grader,
providing ground truth, agent response, and question context for binary correctness judgments.
Grader reliability is validated through spot-checking against human evaluation.

The complete responder prompts are detailed below.

### A.4 Data Construction Pipeline

**Table A1: Data Construction Pipeline: Step-by-Step Example**
| Step | Component | Example Content |
| --- | --- | --- |
| Step 1 | Target Entity A | Hornussen (Swiss team striking sport) |
|  | Distractor B | Baseball (globally popular team bat-and-ball sport) |
| Step 2 | Shared Attributes | Team-based striking game; offense/defense alternation; players take turns hitting; projectiles reach very high speeds ($>$100 mph); protective gear required; governed by formal associations or leagues. |
|  | Distinctive Attributes | Hornussen: strikes a plastic puck (“Nouss”) with whip-like swing using a long wooden rod; defenders intercept with wooden boards; fan-shaped field $\sim$300m; 18–20 defenders spread in wide formation; scoring depends on distance/landing point. |
| Step 3 | Ambiguous Question $Q$ | “Which team-based striking sport features two sides alternating offense and defense, where individuals sequentially hit a high-speed projectile and teammates coordinate to intercept it in the air? Outcomes depend on whether the projectile is intercepted or lands within the valid playing field. Defense relies on wide positioning and collaboration, all offensive players take turns striking, flight speeds often exceed 100 mph, protective gear is required due to impact risk, and the sport is governed by long-standing associations or leagues.” |
| Step 4 | Contextual Information | – Struck object is a plastic puck, resembling an ice hockey puck.<br>– Striking method uses a whip-like swing with a long wooden rod.<br>– Defenders use wooden boards to block the puck in mid-air.<br>– Field: fan shape, $\sim$300m long, 10–12° angle.<br>– Defensive line: 18–20 players.<br>– Scoring: distance/landing-based. |
| Step 5 | Reasoning Path | $Q$ gives a plausible candidate set (e.g., Baseball vs Hornussen). Adding context clarifies unique Hornussen features (puck, whip swing, fan-shaped field, defensive boards), leading to the unique answer = Hornussen. |

### A.5 Rounds Constraints Results

**Table A2: Performance comparison of models under varying average interaction levels. Metrics include accuracy (%), expected calibration error (ECE, %), and average rounds of interaction (Interaction).**
| Interaction | Accuracy (%) | ECE (%) | Setting |
| --- | --- | --- | --- |
| GPT-5 |  |  |  |
| 1.14 | 14.0 | 71.50 | SCALING |
| 1.76 | 16.0 | 71.54 | SCALING |
| 1.90 | 20.0 | 70.06 | SCALING |
| 4.40 | 24.0 | 63.34 | FORCED |
| 6.10 | 20.0 | 69.02 | FORCED |
| 8.32 | 32.0 | 54.68 | FORCED |
| 9.40 | 40.0 | 48.20 | FORCED |
| 11.56 | 40.0 | 46.86 | FORCED |
| DeepSeek-Chat |  |  |  |
| 0.38 | 10.0 | 77.00 | SCALING |
| 0.74 | 8.0 | 80.30 | SCALING |
| 1.54 | 10.0 | 75.20 | SCALING |
| 5.40 | 20.0 | 62.30 | FORCED |
| 6.68 | 22.0 | 52.60 | FORCED |
| 9.26 | 22.0 | 61.30 | FORCED |
| 10.82 | 16.0 | 66.40 | FORCED |
| 12.62 | 24.0 | 61.20 | FORCED |
| Claude-Sonnet-4 |  |  |  |
| 0.16 | 6.0 | 79.90 | SCALING |
| 0.70 | 4.0 | 80.24 | SCALING |
| 0.78 | 8.0 | 81.84 | SCALING |
| 3.12 | 12.0 | 75.80 | FORCED |
| 4.26 | 12.0 | 76.90 | FORCED |
| 6.10 | 10.0 | 76.00 | FORCED |
| 8.02 | 16.0 | 69.10 | FORCED |
| 10.46 | 18.0 | 68.40 | FORCED |
| GPT-4o-mini |  |  |  |
| 2.00 | 4.0 | 49.50 | SCALING |
| 3.62 | 8.0 | 47.60 | SCALING |
| 2.76 | 8.0 | 33.20 | SCALING |
| 14.50 | 4.0 | 65.50 | FORCED |
| 16.50 | 4.0 | 69.70 | FORCED |
| 14.88 | 2.0 | 62.60 | FORCED |
| 11.18 | 6.0 | 56.10 | FORCED |
| 14.92 | 2.0 | 66.70 | FORCED |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: - It would be helpful for the authors to clarify how many examples in the dataset genuinely requires web search to resolve. An analysis quantifying how many items truly depend on external information would strengthen the paper’s positioning.
soundness: 2
strengths: - The proposed benchmark targets an important scenario: in real life, user queries are often ambiguous or underspecified. To effectively address user queries, models need to know when and how to interact with users to resolve these ambiguities.
- The answers are short and easily verifiable, which reduces subjectivity in evaluation.
- Experiments reveal clear and useful empirical insight that overconfidence and lack of uncertainty awareness is a core weakness.
summary: This work proposes a benchmark for evaluating agents’ abilities to navigate tasks with ambiguities or ambiguous queries. This is an important use case scenario. While the field is working towards fully autonomous agents, it’s also important to think about potential issues of premature commitment, where agents act confidently despite uncertainty. The data is built from 210 bilingual expert-written question-context pairs across nine domains, where each example comes with a yes/no question to distinguish between a correct target and a possible distractor. The authors evaluate 17 models and find that accuracy drops significantly under ambiguity, but improves a lot when agents are forced to ask questions.
weaknesses: - Based on the example in Table 1, the query appears to be something a model could likely resolve using its internal world knowledge rather than through web search. During data construction, there was no verification that these queries require external search to be answered correctly. If that’s the case, the rationale for putting the task in an agentic, search-based setup becomes unclear, especially since prior work has already shown that language models struggle to ask clarification questions.
- There is no human performance baseline that measures: when human users are asked to answer these queries, how often will they answer directly without asking clarification questions? This comparison would help contextualize model behavior and clarify whether current systems truly underperform relative to humans.
- There is no information about how many experts were involved in data creation, which platform they were recruited through, or what their credentials or backgrounds are.
- Important statistics on the dataset are missing: how many items are there per domain? What about language distribution by domain?

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 2
questions: - Does the `with-context` baseline (Table 3) have access to search? Why are the numbers low even with access to the hidden context? Is it because the failure of the search engine?

- How is search implemented? What search engine is used?

- What are qualitative examples of the failures backing up the claims in the paper (e.g., *"strategic information acquisition is a distinct capability varying significantly across architectures"*)?
soundness: 2
strengths: - The paper identifies a gap between current search benchmarks and real-world use cases of search systems.
- To address the above-mentioned gap, the authors construct a new expert-written benchmark that specifically tests the interaction capabilities of the models.
- The developed benchmark is not saturated (~14%) and can be used for evaluating both search and interaction capabilities of the models.
- The authors evaluate both proprietary and open-weight models and identify varying behaviors across different models.
summary: The paper identifies a gap between the settings of the current search benchmarks and real-world interactions. Current search benchmarks focus on fully-specified unambiguous queries and evaluate the search and reasoning capabilities of search agents, while real-world human queries are often ambiguous and require multi-turn interactions for disambiguation. To this end, they propose a new benchmark, consisting of 210 expert-written ambiguous queries, that cannot be answered without asking for clarifying questions. The authors report that the current state-of-the-art models achieve less than 14% accuracy on the benchmark, with overconfidence and lack of strategic interaction being the main causes of failures.
weaknesses: ### Major

**Surface-level analyses**

The analyses of the models is limited to the raw numbers and comparisons across models and ablations. To support the claims and provide explanations of the observed behaviors, a deeper analysis is required. For example: why does increasing the number of interaction turns have different effects on different models?

Additionally, providing qualitative examples can provide intuition and help the researchers better understand the failure modes on the proposed benchmark. Going further, developing a taxonomy of failures can further strengthen the paper, as isolated claims based on the raw numbers are hard to synthesize into generalizable findings.

**Search vs Reasoning/QA vs Interaction**

With the current ablations, it is not possible to separate the effect of the following factors that can potentially impact the final system performance: (1) search module (i.e., search engine), (2) reasoning, QA, and tool-calling capabilities, and (3) interaction capabilities.

What are the proportion of the cases where the system failed due to each of the three factors above? Studying additional ablations in Table 3 can provide more transparecy into the failure modes.

***

### Minor

**Literature review**

- The paper misses literature on real-world human-AI interaction datasets (Wildchat, Chatbot Arena). Specifically, as the paper focuses on real-world ambiguous under-specified queries to search-augmented models, Search Arena, a recent dataset of human interactions with search-augmented LLMs, is highly relevant. Analyzing the ambiguity of natural user-written prompts and using them as part of the benchmark can improve the realism and utility of the benchmark.

- Additionally, the paper does not have citations for the works mentioned in the second part of the Section 2 (Lines 140-144).

**Implementation details**

How is search implemented? What search engine is used?

**Presentation and missing details**

- Algorithm 1 is not properly grounded and explained in text.

- The paper misses explanation of the calibration error (C.E.). There is a brief mention of C.E. in Appendix; however, as it is references multiple times in analysis, it would be helpful to include an explanation in the main body of the paper.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: N/A
soundness: 3
strengths: Addresses a clear gap in current search benchmarks- interactivity. The paper makes a compelling argument that real-life search involves this iterative refinement process, and the paper makes a step in that direction.
summary: Paper introduces a benchmark for testing search and dialogue agents on their ability to recognize and resolve *ambigious* queries. Each benchmark data instance contains (1) an ambiguous question, (2) a context string that contains distinctive attributes that are only revealed to the search agent under simulated interaction via another agent if the search agent asks for it, (3) and a single umambigious correct answer. 210 tasks total, bilingual, across 9 domains. 

The dataset is deliberately constructed to be ambiguous through annotator instruction. Across 17 models tested, accuracy remains very low even for SOTA systems. 

The paper represents a solid contribution towards more interactive (and realistic) search benchmarks.
weaknesses: Ecological validity of benchmark construction. Annotators are deliberately asked to construct queries while starting from a target answer. It remains an open question if the scoring well on the benchmark would represent meaningful and demonstrative improvement in real life search tasks. The synthetic interaction channel by forcing a yes/no response from the responder who has the context, is a strange and not fully justified design decision. A more realistic scenario could have the responder respond in free-text (i.e. a search revealed ambiguity, which then revealed a related memory of the object, such that the responder agent could be representative of memory), though controllability would remain harder in that instance. 

The paper also misses out on some heavily relevant related work, notably in the "tip of the tongue known-item search" domain. i.e. BLUR (CH-Wang et al, ACL 2025), Tip of the Tongue Known-Item Retrieval: A Case Study in Movie Identification, Arguello et al 2021, etc.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 2
questions: 1. When a query contains too many constraints, it tends to become excessively long, which introduces a gap from real user scenarios. In InteractComp, is there an emphasis on shorter queries? For example, have you considered questions that do not include all constraints at the first user query but instead require the agent to actively identify ambiguity and ask for clarification?

2. In the data construction process, which parts involve human participation and which rely on LLM-as-Judge? This distinction is important and should be explicitly clarified in the main text.

3. How do you ensure that the ambiguity in each question reflects real semantic ambiguity rather than artificial lexical ambiguity introduced by human phrasing?

4. According to Figure 1(a) in the paper, has the latest model already achieved ~70% accuracy on BrowseComp?
soundness: 2
strengths: 1. This paper demonstrates a good motivation, highlighting that human search behavior is typically iterative, beginning with ambiguous queries and progressively refining them through interaction. This perspective makes the evaluation setting more closely aligned with real-world scenarios.

2. The data construction is based on an insightful idea: ambiguity arises when similar entities share overlapping attributes. The benchmark also considers domain generalization and includes data quality verification, which strengthens its credibility.

3. The evaluation covers a wide range of models, including both proprietary and open-source LLMs, under different configurations and model sizes, providing a comprehensive comparison and analysis. The final conclusions attribute model failures to overconfidence and reluctance to interact, which is well-supported and reasonable.
summary: 1. This paper introduces InteractComp, a benchmark designed to evaluate whether search agents can recognize and resolve ambiguous user queries through strategic interaction.

2. The InteractComp comprises 210 expert-curated questions, which are annotated by humans using an entity-pairing approach and a two-stage quality control protocol.

3. Experiments on 17 models show that even the strongest models perform poorly mainly due to overconfidence and underuse of interaction.
weaknesses: 1. This paper can essentially be understood as targeting a multi-turn collaborative/conversational search scenario. Although InteractComp is verifiable, its query format (as exemplified in Table 1) does not show a clear distinction from complex search benchmarks like BrowseComp. The core design of BrowseComp involves multiple constraints, many of which are themselves ambiguous. Evidently, both benchmarks inherently possess interactive characteristics.

2. Although I find the heuristic approach for data construction quite insightful, the decision to restrict simulated user interactions to yes/no clarifications only seems rather narrow, especially given the goal of modeling realistic user-agent interaction.

3. The human annotation procedure lacks sufficient detail, making it unclear how well the constructed data aligns with real-world distributions. For example, the principles guiding human experts in selecting entities and attributes remain unspecified.