Search Arena: Analyzing Search-Augmented LLMs
Authors: Mihran Miroyan
Tsung-Han Wu
Logan King
Tianle Li
Jiayi Pan
Xinyan Hu
Wei-Lin Chiang
Anastasios Nikolas Angelopoulos
Trevor Darrell
Narges Norouzi
Joseph E. Gonzalez
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Web Search
Human-AI Interaction
Primary area: datasets and benchmarks
TL;DR: We introduce the first large-scale human-preference dataset of user interactions with search-augmented LLMs and provide in-depth analyses.

Abstract: Search-augmented language models combine web search with Large Language Models (LLMs) to improve response groundedness and freshness. However, analyzing these systems remains challenging: existing datasets are limited in scale and narrow in scope, often constrained to static, single-turn, fact-checking questions. In this work, we introduce \textbf{Search Arena}, a crowd-sourced, large-scale, human-preference dataset of over 24,000 paired multi-turn user interactions with search-augmented LLMs. The dataset spans diverse intents and languages, and contains full system traces with around 12,000 human preference votes. Our analysis reveals that user preferences are influenced by the number of citations and types of cited sources, even when the cited content does not directly support the associated claims, uncovering a gap between perceived and actual credibility. To assess cross-setting performance, we conduct cross-arena analyses by testing search-augmented LLMs in a general purpose chat environment and conventional LLMs in search-heavy settings. We find that web search does not degrade and may even improve performance in non-search settings; however, the quality in search settings is significantly affected if solely relying on the model's parametric knowledge. We open-sourced the dataset to support future research.

Published as a conference paper at ICLR 2026
SEARCH ARENA: ANALYZING SEARCH-AUGMENTED LLMS
Mihran Taroog Wu; Logan King, Tianle Li, Jiayi Pan, Xinyan Hu Wei-Lin Chiang, Anastasios N. Angelopoulos, Trevor Darrell, Narges Norouzi
Joseph E. Gonzalez University of California, Berkeley
ABSTRACT
Search-augmented language models combine web search with Large Language Models (LLMs) to improve response groundedness and freshness. However, an- alyzing these systems remains challenging: existing datasets are limited in scale and narrow in scope, often constrained to static, single-turn, fact-checking ques- tions. In this work, we introduce Search Arena, a crowd-sourced, large-scale, human-preference dataset of over 24,000 paired multi-turn user interactions with search-augmented LLMs. The dataset spans diverse intents and languages, and con- tains full system traces with around 12,000 human preference votes. Our analysis reveals that user preferences are influenced by the number of citations, even when the cited content does not directly support the attributed claims, uncovering a gap between perceived and actual credibility. Furthermore, user preferences vary across cited sources, revealing that community-drive and static encyclopedic sources not always appropriate and reliable. To assess performance across different settings, we conduct cross-arens analyses by testing search- augmented LLMs in general-purpose chat environment and conventional LLMs in search-intensive settings. We find that web search does not degrade and may even improve performance in non-search settings; however, the quality in search settings is significantly affected if solely relying on the model's parametric knowledge. We open-sourced the dataset to support future research.
platforms are generally preferred
Search Arena
Code
Dataset
INTRODUCTION
Large Language Models (LLMs) have become popular interface for human-Al interaction, sup- porting information seeking and task assistance through natural, multi-turn dialogue. However, the capabilities of these models are constrained by their reliance on static training data, which prevents them from effectively handling time-sensitive questions, emerging topics, or niche domains. Search- augmented LLMs aim to bridge this gap by retrieving and using live web data during inference. and factually verifiable
Access to search enables LLMs to provide up-to-date, domain-spectie responses (Hilton et al., 2021: Li et al., 2023a). Recent developments (Gemini, 2024: OpenAl, 2024; Perplexity) also reflect the growing interest in scarch- aughehhu LLMs.
Despite rapid progress in developing search-augmented L.LMs, our understanding of how users interact with these systems- what they ask, how they engage in multi-turn dialogue, and what they expect in -iiinn--emmaiss limited. Existing datasets capture interactions with either standalone LLMs (Chiang et al., 2024; Zhao et al., 2024: Zheng et al., 2024) or traditional web search engines (Chen et al., 2024b; Craswell et al., 2020). However, search-augmented LLMs represent hybrid interface different from both: they not only retrieve information through web search but also rely their reasoning and conversational capabilities. The most widely used datasets for evaluating these systems ( (Wei al., 2024) and BrowseComp (Wei al., 2025)) primarily consist of single-turn, monolingual, fact-based queries and are relatively small in scale (typically 5:5k queries). As shown in Figure 1, fact-checking accounts for only one-fifth of real-world user queries; the majority of user prompts, such as seeking analyses, recommendations, or problem-solving guidance, require
Equua contribution.

---

Published as a conference paper at ICLR 2026
Table I: Comparison of Search Datasets. Unlike prior datasets such as SimpleQA Wei et al. (2024) and BrowseComp Wei al. (2025), which are static, English-only, single-turn fact-seeking queries, Search Arena evaluates models in diverse, open-ended, multilingual, and multi-turn settings. We release 24,069 conversations with 12,652 preference votes. Further analyses are provided in Section 2.
Dataset
#Cons #Langs Mulliturn Answer/Judge
Conversation Properties
Metadata
SimpleQA
4,326
I (FNN
No
Short ground truth Expert- short factual queries
Verified supporting URLs topic tags Topic tags
BrowseComp
1,266
(EN)
No
Short ground truth Expert-written challenging prompts
with detailed constraints
Search Arema 24,069
71
Yes
Human preference open-ended, crowd-sourved prompts Retrieved URLs, full model across diverse intents and topics
user intent, and topic tags
Factual Lookup
Switch 2 price
15.4%
Info Synthesis [|
Summarize top privacy-related news incidents, legisiations reported in January 2025...
Ineed represent chronicle unified
300
Analysis eo
(udm) python. research the chronicle udm fleld
Count
10.9%
0.0%
9.1%g
best running shoes beginners that affordable
Recommendation
Explanation
Introduce Denver's history to non US
Creative Generation
satirical encyclopedia article Salvador Borrego
Guidance Ø
How do block domain but allow specific branch uBlock Origin?
Fartuall Lochao
Brio Sprthead
Analeco
Pecemnnndlatoo
Ensiaretted
Creative dentratien
Guideecc
Text Processing 9
Could you summarize ewebsite>?
Figure I: (Left) Nine intent categories with representative examples (truncated), In-the-wild prompts often ambiguous and require real-time web retrieval. (Right) Intent distribution across prompts. Most queries require more than a simple fact lookup. ranging from information synthesis to creative generation. The Other category is excluded.
combination of factual retrieval, reasoning, and open-ended dialogue. User expectations also extend beyond factual correctness: preferences can be shaped by the number, relevance, and credibility of citations, as well as the presentation style of responses.
To address these gaps, we crowd-sourced the first large-scale human-preferrnce dataset of in-the-wild user interactions with search-augmentdd LLMs. We developed Search Arena, an open evaluation and data collection platform that presents anonymized side-by-side model outputs in multi-turn settings and collects human votes. During the seven-week deployment period, we gathered and publicly released 24,06 ,,69 conversations, along with 12,652 paired preference judgments. The dataset spans 11,650 users across 136 countries, 13 models, around 70 languages (including 1 11% multilingual prompts), and over 5,000 multi-turn interactions. We also introduce a user intent taxonomy in the context of seareh-enabled human-AI interactions. As detailed in Table I and Section 2, Search Arena provides a broad coverage across linguistic and intent features.
We not only analyze user prompts to search-augmented LLMs, but also their preferences. We model user preferences through the Bradley-Terry model (Bradley & Terry, 1952; Chiang et al., 2024; Tianle Li, 2024) and study how different response characteristics interact with user judgments. We find that reasoning, a larger search context window, and responess are positively associated with user preferences. Since citations are central to trustworthy web-grounded outputs, we also examine citation features. Our results show that users prefer responses with more cited sources (Figure 4) and those citing tech-related platforms, community blogs, and social networks, but less Wikipedia (Figure 6). While correctly attributed citations positively interact with preferences (8 0.285), we observe a positive association with irrelevant citations (8 = 0.273). This raises concerns that users may be overly influenced by citation presence, even when they do not support the associated claims.
responses
We also investigate how saarch-augmented and non-search models perform across different settings by deploying a non-search LLM in the Search Arena, and a search-augmented LLM in general- purpose Text Arena. Our results show that conventional L.MMs underperform in search-intensive
2

---

Published as a conference paper at ICLR 2026
settings (p-value 0.009). However, search-augmented models perform comparably in general chat settings, with improved performance on factual lookup queries (p-value 0.012) and slightly degraded performance text processing prompts (p-value 0.077).
In summary. our contributions are follows: (1) we release the first large-scale human-preference dataset of 24k user conversations with earch-auggmettd LLMs,, along with 12k preference votes, system metadata, user intents, and prompt topics; (2) present the first analysis of how different characteristics of pearch-augmenter LLMs interact with human preferences; and (3) we conduct the first cross model in Text Arena, reporting that web search augmentation does not hurt and may improve performance across settings, while the internal parametric knowledge of models alone is not sufficient in search-intensive settings.
evaluation by testing non-search model in Search Arena and search-augmented
2 HUMAN PREFERENCE DATASET IN SEARCH
We launched Search Arena, an open, crowd-sourced evaluation platform for search- sarchhaaugmeted LLMs, on March 18, 2025. The Search Arena is implemented as separate tab within the Chatbot Arena web application (Chiang et al., 2024), where users interact exclusively with search-enabled models. The search mode interface encourages During each session, two anonymous models respond to a user query, the user can then cast vote for their preferred model response. Details on user interface and potential limitations are reported in Appendix A and Section 5.
search-intensive queries, users adjust their expectations.
Between March 18 and May 8, we collected more than 24,000 conversations and 12,000 user votes across 13 models, spanning range of model configurations (e.g., reasoning models, search context sizes, etc.). The collected dataset contains model identities, the user vote, conversation histories, and system metadata (e.g., reasoning traces, retrieved URLs). Table presents key differences between Search Arena and prior datasets (SimpleQA (Wei et al., 2024) and BrowseComp (Wei et al., 2025)), including dataset scale, prompt characteristics, and available metadata. In the following subsections, we analyze prompt distributions across linguistic and intent dimensions in comparison to existing benchmark datasets.
English
Search-rrna
Binese Comp
Chirese
7.00
10
SlimplGA
German
Search-Amm maxet 37
Spanish
Browse Comp meam 1033
Japanese
2.5%
SimpleGA mean: 16.3
2.7%
Deneky
Korcan
2.8
Portuguese
2.2%
Italan
Indonesian 0.00
Polish 0.7%
Turkish
0.5%
Prompt Length (words)
Count (log scale)
Figure 2: (Left) Search Arena prompt language distribution. The dataset is multilingual, spanning over 70 languages, with English prompts accounting for 58.3% of the data. (Right) Prompt length distribution of Search Arena (blue), BrowseComp (purple), and SimpleQA (green). Search Arena prompt lengths are more spread out and cover the range of BrowseComp Wei et al. (2025) and SimpleQA Wei et al. (2024) questions.
2.1 LINGUISTIC AND CONVERSATIONAL DIVERSITY
Search Arena was collected from 1 11,650 users across 136 countries, resulting in substantial linguistic and geographic diversity. The prompts span over 70 languages, with 30 represented by least 10 conversations. English accounts for 58.3% of the data, followed by Russian (1 (118%) and Chinese (7.0%). More than 1 11% of the prompts are multilingual. Figure (Left) shows Search Arena prompt language distribution across the top 15 languages.
Since the platform supports multi-turn interactions, 22.4% of conversations in the dataset are multi- turn, typically clarifications follow-up queries. Furthermore, shown in Figure 2 (Right),
3

---

Published as a conference paper at ICLR 2026
SimpleQA prompts are short and fact-oriented (16.3 words average), while BrowseComp prompts are intentionally constructed to be long and constraint-heavy (103.3 words on average). In contrast, Search Arena includes both brief, under-specified queries well as longer, detailed requests (57.1 words on average). Additional details and analysis of linguistic features are provided in Appendix B.
2.2 INTENT DIVERSITY
Existing search-augmented LLM evaluation datasets focus solely on factuality. To study how in-the- wild user prompts from Search Arena differ from SimpleQA (Wei al., 2024) and BrowseComp (Wei et al., 2025) questions, we apply an LLM-based dataset differencing framework (Dunlap et al., 2024; Zhong et al., 2022). We also compare Search Arena prompts with the prompt distribution of Text Arena (Chiang et al., 2024), where users' expectations of the models are not influenced by search settings. GPT-4.1 is used for generation and GPT-4, 1-mini for ranking of candidate distinguishing properties; more details on the pipeline are provided in Appendix C. The summaries of the top properties reveal high-level pairwise differences between the datasets:
Search Arena vs SimpleQA: Search Arena prompts are broader and more complex, often requiring analysis, synthesis, or creative generation, while SimpleQA prompts are narrowly focused on retrieving specific, factual information with minimal context or interpretation.
Search Arena BrowseComp: BrowseComp prompts structured as investigative chal- lenges, often requiring synthesis of clues under specific constraints, whereas Search Arena prompts prioritize immediate, functional assistance.
Search Arena vs Text Arenu: Search Arena prompts focus on real-world factual lookup and decision-making support, often involving up-to-date information. In contrast, Text Arena prompts focus on problem-solving. programming help, and creative generation.
These summaries show high-level differences between prompt distributions across three settings- user prompts to search-augmented L.MSs (Search Arena), static factuality questions (SimpleQA and BrowseComp), and prompts to regular L.LMs (Text Arena). We also study Search Arena prompts through topic modeling and observe diverse and real-time topies, ranging from market analysis to health discussions (see Figure B.4).
For a structured and in-depth analysis, we introduce a taxonomy of user intent categories. While prior work has explored user intent classification in general dialogue settings (Liu al., 2024b; Shah et al., 2023), we focus on real-world interactions in search-augmented chat-based settings. The taxonomy includes nine categories: Factual Lookup. Information Synthesis, Analysis, Recommendation, Expla- nation, Creative Generation, Guidance, Text Processing, and Other. We use secondary labels for ambiguous or multi- purpose prompts. Details on the taxonomy design, as well as descriptions of cat- egories, are provided in Appendix B. We scale the annotation to the full dataset using 1PT-4.1, with a manually tuned prompt seeded on 100 examples and validated on 150 multilingual prompts. The resulting Cohen's 's kappa of 0.812 indicates strong agreement between model- and -anotttdd labels. The annotation pipeline and validation details are provided in Appendix B.
Figure 1 shows the resulting intent distribution, along with examples for each category. Factual lookup queries account for only 19.3% of user prompts. The remaining prompts require higher-order capabilities, such as synthesis, guidance, or analysis. We also analyze how linguistic features vary across intents; specifically, average), whereas the remaining set is associated with longer, more complex queries (66.7 words on average). Further analyses of intent categories are provided in Appendix B.
find that factual lookup prompts typically shorter (17.2 words on
We report comparative analysis with additional related datasets (CORAL, WildChat) in Appendix G.
3 PREFERENCE ANALYSES IN SEARCH
With over 12,000 anonymized human preference votes, Search Arena supports fine-grained analysis of how different response features interact with user preferences. We report model performance through both head-to-head win rates as well as scaled coefficients estimated using the Bradley-Terr Terry model (Bradley& & Terry, 1952; Chiang et al., 2024) in Appendix D. To analyze how features interact with user preferences, we follow prior work (Tianle Li, 2024) by adding normalized differences between pairwise features to the Bradley-Terry model and reporting the fitted coefficients.
4

---

Published as a conference paper at ICLR 2026
A
R
User Manchester United Transfer news
Note: Manchester United soccer club
Who the richest person allve?
"Reasoning" Bearch-Augmented LLM
Model
The "Winner" Model
<think>... The Forbes Real-Time Billionares
.e.Rennt Signings:
..Macchester United are prioritizing major overhaul of their attacking options...
list from [1] mentions Elon Musk at the top with $298 billion, followed...
Joshua Zirkzee: On July 14,
2024, Manchester United secured the services of Dutch striker Joshua Zirkzee from Bologna for 236.5 million... [1][tttps::/ en. wilkipedia..rg//kkii
Another source, Investopedia (4), mentions Musk at S316B as of April 2025, but the Forbes India article [7] had an error and couldn't used. The Wikipedia page [3] has older dats, it's less relevant...
Hugo Ekitke (Eintracht Frankfurt): The 22-year-old French striker has scored 19... [1](tttps://
www.nbcspont www..nnnssosssco//sseeooo news/...) + It's in May 2025
Forbes 2025 list [5] from April 5th is likely the most current...
Joshua Zirkzee)....
Therefore, the consensus across multiple sources April 2025 paints t/t.n//think>
For news related questions, wikipedia data can be outdated.
Figure 3: (Left) Reasoning trace example, containing multi-document analysis, filtering, and syn- thesis. (Right) Example of a rejected response citing Wikipedia for a sports news question. The preferred response cited the sports division of a news outlet, containing more up-to-date information.
As the correlational analysis in this section relies on user- user-provided votes, we first clarify their subjective nature and assess their reliability. We do not treat the numan-preferre response as the objectively "correct" one, but rather as the response better aligned with the user's personal preferences. In our evaluation and data collection setup, the prompt tied to the judge (i.e., user): this pairing critical, as only the original user knows the hidden intent behind their prompt. To verify that users not voting randomly, we conducted a small study on 100 samples with expert annotators and found expert-user agreement to be 68% when excluding tie votes (random agreement = 50%). Given the inherent subjectivity and the lack of a single "correct" answer for many prompts, some disagreement is expected.
In the following section, analysis is purely correlational rather than causal, and we view more controlled causal investigation as an important direction for future work. We first focus on two key feature groups: (1) general features such model type, search depth, and response length (Subsection 3.1), and (2) citation-related features, such as number of citations, citation sources, and citation attributions (Subsection 3.2). We then study how search and non-search models generalize in search-heavy and general chat scenarios in Subsection 3.3. Additional details and further analyses provided in Appendix D and Appendix E.
3.1 GENERAL FEATURES
Reasoning, Reasoning models generally perform better in Search Arena prompt distribution, with the top three models achieving over 60.0% average win rates, suggesting that reasoning improves performance in cearch-augmente we observe prompt analysis, problem decomposition, and backtracking behaviors in reasoning traces. Additionally, we find that reasoning models not only interpret and analyze retrieved content but also rerank sources and filter out irrelevant information. One example is in shown Figure (Left).
chat interactions. Consistent with prior work (Gandhi et al., 2025),
Search Context Size. Models with high search context windows outperform those with smaller search context. For sonar-pro, the version with high search context has a higher (p < 0.01) average win rate (63.9%) compared to the model with medium search context (57.6%). However, the difference is not significant for GPT-4o models with medium and high search context sizes. This finding indicates that models with higher search context retrieve more web sources, leading to more preferred responses; we analyze the effect of citations in Subsection 3.2.
Response Length. Consistent with findings from prior work (Chiang et al., 2024; Steyvers et al., 2024; Tianle Li, 2024), we observe that users are biased towards more verbose responses. The Brarley-Terry coefficient corresponding to response length is positive and statistically significant (Beaghh = 0.334), indicating that users tend to prefer longer answers. The positive correlation between model score and response length is also shown in Figure 4 (Left). Additionally, Figure 5 (Left) shows the length distribution across eight user intent categories; responses to Factual Lookup prompts much shorter (168.3 words on average) compared to Creative Generation (422.8 words) and Analysis (393.2 words) prompts. Furthermore, find that the response length coefficient on Factual Lookup
5

---

Published as a conference paper at ICLR 2026
prompts (Crength, factual 0.156) is 2.14 times smaller than the effect the full dataset, suggesting that users prefer less verbose answers to Factual Lookup queries compared to other categories.
3.2 CITATION FEATURES
Citations are central to the trustworthiness of reb-grounded outputs in search- scenarios and a unique feature of Search Arena compared to Text Arena (Chiang et al., 2024). Therefore, we further examine how these factors influence user preference along three dimensions: number of cited sources, types of cited sources, and attribution of inline citations.
Number of Cited Sources. We find a positive and statistically significant coefficient for the number of citations (Benstions = 0.209), indicating that users favor responses with more references. The positive association between model score and response length is shown in Figure 4 (Right). Furthermore, we observe that reasoning models cite fewer sources than non-reasoning models with similar con- figurations, consistent with our earlier observation that reasoning models filter irrelevant content. Unsurprisingly, models with high search context size end up citing more sources in their final re- sponse. Additionally, Figure (Right) shows the distribution of citation counts across prompt intent categories. Notably, responses to Factual Lookup prompts contain fewer citations (5.7 average), compared to Recommendation (6.9 citations on average) and Info Synthesis (6.8 citations on average) prompts, due to the broader web coverage needed for the latter.
Types of Cited Sources. We categorize retrieved URL domains into nine groups (e.g., news, Wikipedia, social media, tech/code platforms, etc.); categorization details appear in Appendix E. Figure 6 (Left) shows source category coefficient estimates with 95% confidence intervals; we also control for citation counts to account for the bias shown earlier. Citing tech-related platforms (e.g.. Stack Overflow), community platforms (e.g.. Substack), and social media (e.g.. TikTok) positively associated with user preferences with fitted coefficients equal to Buch = 0.073, Severunity = 0.061, and /Bacial 0.057, respectively. Surprisingly, citing Wikipedia is negatively correlated with user preferences (,3wu = -00071). To interpret the latter result, we inspect rejected model responses citing Wikipedia and identify two potential explanations: (1) Wikipedia articles are often very lengthy and broad, not directly relevant to a user's s question, and (2) citing Wikipedia is not preferred for queries requiring real-time information. A qualitative example appears in Figure (Right).
Citation Attribution. We then study how the correctness of iitation-to-clain attribution (i.e., whether the inline citation supports the attributed claim) interacts with user preferences. Formally, for each multi-turn interaction, we decompose model responses into set of claim-citation pairs (c,, u)) where denotes a textual claim, and u, is the corresponding inline citation. For each pair, we evaluate whether the webpage content D, supports, is irrelevant to, or contradicts the claim es. This process is automated via an LLM- ---added pipeline described in Figure 7 through an example. Due to scraping challenges and the high cost of I.M calls, we run the pipeline on roughly 100 conversations per intent category (800 examples in total). The resulting output of each conversation is a set of triplet {(a,, us ts)s ) where t; E { (Support, Irrelevant, Contradict) and N is the total number of claims per conversation. We then compute the number of supporting, irrelevant, and contradicting claims per
1300
0.378
$45.70
17.192 939.23
1150
Store Salmata
Stone batimate
1100
1050
1030
1002
1030
950
Average Resparoe Length (Worda)
Avenage Number of Clertons
Figure 4: (Left) Positive relationship between model score and average response length. (Right) Positive relationship between model score and average number of citations.

---

Published as a conference paper at ICLR 2026
Creative Generation
Recommencat.on
Guidence
Info Synthesis
Anelysic
Analysis
Recommendation
Guidance
Info Synthesis
Explanat on
Exolanation
Factual Lookup
Text Processing
Generation
Factual Lookup
Teat Pronessing
Response Length (words)
Citation Count
Figure 5: (Left) Response length distribution across user intent categories. Responses to Factual Lookup prompts more concise (168.3 words average) compared to other categories. (Right) Citation count distribution across user intent categories. Responses to Recommendation (6.9 on average) and Info Synthesis (6.8 on average) prompts contain more citations compared to Factual Lookup (5.7) and Text Processing (4.2) prompts.
count
tech coding
community blog
media
youtube
academic
ineevvan count
us news
edu
foreign news
wikd
-0.1
0.1
0.2
0.3
0.4
Coefficient Estimate
Coeffictent Estimete
Figure 6: Citation Control. (Left) Bootstrapped coefficient estimates of citation features. (1) Users prefer responses with more citations. (2) Citing tech-related and community platforms, as well as social media is positively associated with user preferences. (3) Citing Wikipedia negatively interacts with user preferences. (Right) Bootstrapped coefficient estimates of the number of supporting, irrelevant, and contradicting claim-citation pairs. The number of supporting and irrelevant pairs is positively correlated, while the effect of contradicting pairs not significant.
model response and add them as control covariates in the Bradley-Terry analysis. Implementation details, including scraping tools, parsing logic, and validation process, provided in Appendix E.
Figure 6 (Right) shows bootstrapped coefficient estimates for the number of supporting, irrelevant, and contradicting claim-citation pairs. While users tend to favor responses with more citations, as shown in Figure 6 (Left), the number of contradicting claim-citation pairs does not show significant effect on user preference. Furthermore, both supporting (Sumpeat = 0.29) and irrelevant (Buelevam 0.27) claims are positively correlated with user preference. Thus, users do not distinguish between supporting and irrelevant citations and generally prefer more citations, even if the citations do not directly support the claims. Upon inspection, in irrelevant citation cases, models may fabricate connections, cite tangentially related sources, or present inferred claims that subtly deviate from the source content. This finding suggests that users may be influenced by the mere presence of citations, rather than their proper attribution to generated claims. We raise this as an open issue for the community: improving citation attribution is critical to ensure that citation- heavy responses are not misperceived as factual and trustworthy.
3.3 SETTING ANALYSIS
Search Arena evaluates models where user prompts and expectations are conditioned on models' access web search. In this section, we study how search and non-search models perform under var-
7

---

Published as a conference paper at ICLR 2026
(a) Claim-Citation Breakdown Pipeline
(b) Analysia on Citation Attribution
User
Gj Sesson of "Grey's Analomy" premiered September 26, 2024, and
How many episodes of Grey's Anatomy has Ellen Pompeo appeared in this season?
2024,
44 https://ttkkppeeet hrtpppcdda.rpwiiii/Epeeeedd Gray%
Bearch-Augmented LLM
4 Contradict
Controlled
Ellen Pompeo appeared in seven episodes during Season 21 "Grey's Analomy." This
Despite her limited on-screen appearances, she provided voiceover narration for each episode. com-elien-pompeo httpepppl.cc//llleee.eeee. comvellen pompeo- set appear...
#ouppot
medevat
season, which promiered on September 26, and concluded November 2024, consisted of 18 episodes [1]
htps:/ppople.co
#Overuder
2 Support
Coefideer Extrnate
Despite her limited on-screen appearances, continued to
executive producer and provided voiceover narration for each episade [2].
II "Clain"
LLM
"AssTeg": cSupport. Contradict Izzelevans>
[1]: hitpe://it wikizedia. erglwitbppieodi.cm di... [2]: https//ppeppl comvellan-pomppe-eee... Scraping
Grey's Anatomy episades (Grey's Anstomyy).. an American...
"Reason": "The centent' eplaode list ahovs episcdes airing until May 15, 2025, which contradieta the claix. that the aeason
concluded cn Novenber 22, 2024."
Figure 7: Experimental Setup for Citation Attribution Analysis. (a) For each multi-turn conversa- tion, we retrieve the cited web content and use an LLM-based pipeline to decompose each model response into individual claims, followed by citation attribution labeling. (b) For each claim-citation pair (es, u. 4), we compute turn-level citation counts across the three categories and add correspond- ing features the Bradley-Terry model. Results are shown Figure (Right).
ious prompt distributions and user expectations--pecifeeee results on model performance across benchmarks are provided in Appendix F.
Search Arena Text Arena. Additional
To investigate performance differences across settings, we deployed Gemini-2.5 Pro Experimental (Gemini, 2025)- and without access web search-to the Search and Text Arenas. In the Text Arena, the search model only competed against its non-search version; in the Search Arena, the non-search model competed against all other supported search models. Additionally, inline citations were disabled for the search model in the Text Arena to avoid vote bias. In the Text Arena, users typically assume that models operate in closed-book settings without access external information, while in the Search Arena, user expectations are explicitly conditioned on the search setting. This setup enables us to examine whether and under what conditions access to web search enhances or degrades model performance.
Text Arona Suttings
Search Arcna Scttings
Co
Fortust Loologg
Trio Systheds
Experration
Craaiit Genorazzo
Gullantca
Taxt Prevtriing
Fatbust Lootug
Sero Syrteeii
Explrnaiio
Creative Genurazon
Geslannc
Text Preuediin
Figure Cross-Arena Vote Distribution across Text Arena (Left) and Search Arena (Right) broken down into user intent categories. Users prefer the search model for Factual Lookup and Info Synthesis queries in both settings and the nen-search model for Text Processing queries in Text Arena.
Text Arena Setting, We collected 544 battles between the search and non-search versions of the model, yielding 245 ties (45%), 143 search-prefenee (26%). and 156 non-search preferred votes (28%). We observe a high proportion of tie votes, and the difference between search-prefence and ci-ssarcch-prefenee votes is not statistically significant (p-value 0.244). On aggregate, search and non-search models have comparable performance the Text Arena. To further analyze performance differences by prompt type, apply the intent classification pipeline from Subsection 2.2 on the 544 collected Text Arena prompts. The distribution of votes by intent class is shown in Figure (Left). We observe a high proportion of ties in Analysis, Creative Generation, and Guidance queries, indicating that there are no significant differences between model responses for these types of prompts. However, for Factual Lookup (p-value 0.012) and Info Synthesis (p-value 0.095), the difference
8

---

Published as a conference paper at ICLR 2026
is more pronounced in favor of the search model. Thus, the saarch-augmenedd model is preferred for knowledge acquisition tasks- even in the absence of well-defined user expectations because web reb-grounded responses typically provide precise data, statistics, dates, names, and domain-specific terminology. We also note that the difference in performance for Text Processing (p-value = 0.077) queries favors the non-search model. In these cases, the non-search model often provides structured responses (c.g., numbered or bulleted lists, headings), which suggests that presentation style may impact user evaluations.
Search Arena Setting. We collected 315 pairwise battles between non-search model and search- augmented models in Search Arena, with 99 ties (31%), 126 search-preferred (40%), and 90 non- search-preferree votes (29%). The difference between search and non-search-preeferee votes is statistically significant (p-value = 0.009); the non-search model underperforms under a search- conditioned distribution. A detailed vote distribution by user intent is shown in Figure 8 (Right). Compared with the Text Arena (Figure 8 (Left)), tie votes are less frequent across all categories, indicating that the differences between model responses are more pronounced. The difference most expressed for Factual Lookup (p-value = 5.8 x 10 9) and Info Synthesis (p-value 0.092) queries.
These cross-arena experiments demonstrate that search augmentation does not hurt performance in non-search settings and can improve responses to queries related to information retrieval and synthesis. However, removing web search significantly hurts model performance in search settings.
4 RELATED WORK
Large Language Models. LLMs have made impressive advances in language understanding, dialogue generation, and reasoning, enabled by techniques such as large-scale pretraining (Anthropic, 2024; Bai et al., 2023; Brown et al., 2020; Grattafiori et al., 2024; Liu et al., 2024a; Touvron et al., 2023), tfain-of-thouuht prompting (Kojima et al., 2023; Wei et al., 2023; Shinn et al., 2023; Wang et al., 2023; Yao et al., 2023a), and reinforcement learning with human feedback (RLHF) (Bai et al., 2022; Ouyang et al., 2022a). The dataset and evaluation landscape has moved from static benchmarks (Hendrycks et al., 2021a; Joshi et al., 2017: Hendrycks et al., b)2 toward more challenging settings such as deep reasoning (Li et al., 2024; Kazemi al., 2025; Zeng et al., 2024; Lin al., 2025), coding (Jimenez et al., 2024; Jain et al., 2024), and open-ended dialogue under crowd-sourced evaluation (Chiang al., 2024; Zhao al., 2024). While crowd-sourced setups can mitigate concerns over data contamination in pretraining (Yang et al., 2023; Cheng et al., 2025b), recent work shows potential biases and oversights in human preferences (Clark et al., 2021: Wu & Aji, 2025).
As LLMs gain tool-use capabilities (e.g.. APIs, code interpreters, and web browsers) (Hilton et al., 2021; Schick et al., 2023; Grattafiori et al., 2024; Patil et al., 2024; Yao et al., 2023b), domain-specific benchmarks and datasets have emerged to analyze and evaluate L.MSs under different environments (Zhou et al., 2024: Li et al., 2023b; Liu et al., 2023). In the context of web search, several search- augmented LLMs have been developed (OpenAI, 2024; Perplexity: Gemini, 2024), which retrieve live information to support better reasoning. However, existing benchmarks such as SimpleQA (Wei et al., 2024) and BrowseComp (Wei et al., 2025) are limited to single-turn, fact-based, monolingual queries. Although WebArena (Zhou et al., 2024) contains diverse user prompts and web-based interface, it emphasizes closed-world web navigation tasks rather than open-ended search, reasoning, and dialogue. We introduce Search-Arema the first sarge-scale, crowd-sourced dataset for search- augmented LLMs with human preference signals, covering diverse intents, topics, and multi-turn interactions across 70+ languages, collected through transparent, open platform.
Traditional and LLM ILM--nnterated Information Retrieval (IR). Information retrieval is a long- standing task, with early methods like BM25 (Robertson et al., 1994), PageRank (Page et al., 1998), and embedding- based approaches (Monir et al., 2024; Huang et al., 2020). Several static benchmarks (Thakur et al., 2021; Muennighoff et al., 2023) and large-scale web search datasets with user logs and preference signals (Voorhees& & Harman, 2005; Craswell et al., 2020; Chen al., 2024b) have enabled robust evaluation of retrieval systems and comprehensive studies on user behavior.
With the rise of LLMs, information retrieval has moved beyond traditional search and become integrated into LLM workflows. In Retrieval Augmented Generation (RAG) settings, retrieved text is appended to the input prompt (Lewis et al., 2021: Guu et al., 2020; Hilton et al., 2021; Chuang et al., 2025); Datasets for evaluating these LLM-IR systems span needle-ine in- nh-a-baystack retrieval (Kamradt, 2023; Wu et al., 2025), citation attribution (Zhang et al., 2024; Abolghasemi et al., 2024), and general question answering (Yang al., 2018; 18; Han et al., 2024; Wei et al., 2024; 2025). More

---

Published as a conference paper at ICLR 2026
recently, this line of work has been extended from direct single-round information--seking to natural, conversational search scenarios (Gao et al.. 2023; Mo et al., 2025; Cheng et al., 2025a; Adlakha al., 2022), and then to search-augmentee LLM systems operating the open web (Hilton et al., 2021; Schick et al., 2023; Yao et al., 2023b). We view search-augmented L.M systems as a variant of RAG, with clear differences, including (1) "unbounded" search space over the whole open web and (2) multi-round agent-based query refinement, search, and generation. However, for search augmented LLMs, existing benchmarks lack full human-Al interaction traces, including queries, retrieved documents, responses, and preferences, which are needed for human-centrie analysis as done in traditional IR (Craswell al., 2020; Chen al., 2024b). To fill this gap, we introduce Search Arena, large-scale open dataset that enables human-centered analysis of this emerging interface.
LIMITATIONS, CONCLUSION, AND FUTURE WORK
Limitations, Future Work, and Broader Impact. Crowd-sourced data analysis provides valuable insight into real-world user preferences but comes with limitations and broader social implications. First, the collected data may reflect demographic skews and may not be fully representative of the broader population, as not all users choose to vote and human judgments are inherently subjective (Chen et al., 2024a; Clark et al., 2021). Second, because conversational data with human preferences personal and potentially valuable for model improvement (Ouyang al., 2022b), its release requires careful consideration of both privacy and equitable access across the community.
To address these concerns, we anonymized model responses and randomized their left/right placement reduce known human biases. During the data collection period, no early access data was granted to model providers, nor were any pre-release models deployed on the platform. We obtained user consent at interaction time and enforced strict privacy policy. To aid responsible interpretation, we also analyzed known biases (c.g., response length; Subsection 3.1) and reported user demographics in Figure B.1 and Figure B.5. Additional details are provided in Appendix
We identify several limitations in our current study that open avenues for future research. While primary focus is dataset construction and interaction analysis, the Search Arena dataset facilitates the following directions:
Objective metries: As detailed in Section 3, user preferences are often influenced by surface-level features, such as citation counts or source reputation, rather than the intrinsic quality of the cited sources and the generated content. Consequently, user-provided labels may diverge from objective standards like factuality, citation-claim consistency, and source specificity. Future work should expand the evaluation framework to include these objective metrics and analyze their correlation with human preference signals to better understand the gap between perceived credibility and actual groundedness.
Our current evaluation relies on an online which, while robust, restricts the scalability of testing new models. Future work can leverage our dataset to construct automated benchmarks for offline evaluation. This could involve training reward models or L.LMs that approximate human preferences, potentially combining them with the objective metrics discussed above to create comprehensive scoring rubric.
Offline evaluation:
-nte-ii--tttoopppop setup,
Model development: Our cross-arena and preference analyses reveal that blindly augment- ing models with search is not always optimal. Future research can utilize these insights to improve both the search module (e.g., better source filtering) and the backbone LLM (e.g.. optimizing policies for when to trigger search versus relying on parametric knowledge).
Conclusion. We present Search Arena, the first large-scale dataset and analysis of human interactions with search-augmented LLMs. Spanning over 24k multi-turn conversations and 12k human preference votes across more than 70 languages, the dataset captures a broader range user intents and topics than prior benchmarks. Our analysis reveals that user preference is positively associated with citation count and certain source types; however, models do not always cite correctly, highlighting a key challenge in trustworthy systems. The cross-arena experiment further shows that search-augmented and non-search models behave differently across settings. We release the full dataset to support future research in search-an aageh-a LLMs and human-centric analyses.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We are deeply grateful to Lisa Dunlap for her invaluable feedback and thoughtful discussions. Sky Computing Lab is supported by gifts from Accenture, AMD, Anyscale, Cisco, Google, IBM, Intel, Intesa Sanpaolo, Lambda, Lightspeed. Mibura, Microsoft, NVIDIA, Samsung SDS, and SAP. Authors, part of their affiliation with UC Berkeley, were supported in part by the National Science Foundation, US Department of Defense, and/or the Berkeley Artificial Intelligence Research (BAIR) industrial alliance program, as well as gifts from Amazon.
REFERENCES
Amin Abolghasemi, Leif Azzopardi, Seyyed Hadi Hashemi, Maarten de Rijke, and Suzan Verberne. Evaluation of attribution bias in retrieval- -urmmeet-- large language models, 2024. URL https:
//arxiv. wtgg 2abb//2411 12380.
Vaibhav Adlakha, Shehzaad Dhuliawala, Kaheer Suleman, Harm de Vries, and Siva Reddy. Topi- oequ: Open- conversational question answering with topic switching. Transactions of the Association for Computational Linguistics, 2022. ISSN 2307-387X. doi: 10.1162/tacl a 00471.
URL www.tttpp//.cc /doi. org/ 11..0.1112/aaa.. 00411. 10
Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. URL httpt:///wwcccch cdn. anthropic, a. com / de a 9b01 c9ab7 (ataabdf333300bbbbee 218.0000000000111888777777 8857687 6Mod/d
Card_Claude
pdf. 9
Jinze Bai, Shuai Bai, Yunfci Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge. Yu Han, Fei Huang, et al. Qwen technical report, 2023. URL https: httppp////rre http///xxvvvvr//// org//brsg/ss////
2309.16609, 9
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage. Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL https::///......... /arxitii v. orgg abs 2204. 05862. 62.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 1952. 2, 4, 29
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 2020. 9
Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or LLMS as the judge? study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2024a. URL ht https: http:///aaaaathooooooooo
org/ 2024. emnlp-ma mam...main....m..mmmmmmmmmmmmmmm 474/. 10
Qi Chen, Xiubo Geng, Corby Rosset, Carolyn Buractaon, Jingwen Lu, Tao Shen, Kun Zhou, Chenyan Xiong, Yeyun Gong, Paul Bennett, et al. Ms marco web search: A large-scale information-rich web dataset with millions of real click labels. In Companion Proceedings of the ACM Web Conference 2024, 2024b. 1, 10
Yiruo Cheng, Kelong Mao, Ziliang Zhao, Guanting Dong, Hongjin Qian, Yongkang Wu, Tet- suya Sakai, Ji-Rong Wen, and Zhicheng Dou. CORAL: Benchmarking multi-turn conver- sational retrieval-augmenntd generation. In Findings of the Association for Computational Linguistics: NAACL 2025. Association for Computational Linguistics, 2025a. URL https: //aclanthology .rg. 2025. findings-naace 72/. 10, 36
Yuxing Cheng, Yi Chang, and Yuan Wu. survey on data contamination for large language models, 2025b. URL htt.ttttt///xiiiiii wttp://///////s//eee 4425. 9

---

Published as a conference paper at ICLR 2026
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos. Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference. In Forty-first International Conference on Machine Learning, 2024. URL httph:ttt//ccc..ctt /opprr//o/eeevveeeeeeee ew, net/ forum? ? i=d=
3MW8GKNYZI. 2, 4, 5, 6, 9, 29
Yung-Sung Sung Chuang. Benjamin Cohen-Wang, Wang, Shannon Zejiang Shen, Zhaofeng Wu, Hu Xu, Xi Victo- ria Lin, James Glass, Shang-War Wen Li, and Wen tau Yih. Selfcite: Self-supervised alignment for context attribution in large language models, 2025. URL httpp::/aaxxii.org//aaaa 2502. 09604. 9
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. All that's 'human' is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguisties and the / 11th International Joint Conference on Natural Language Processing (Volume Long Papers). Association for Computational Linguistics, 2021. URL
.rrg/ 2021.0222200 acl -conog..........
565/. 10
Nick Craswell, Daniel Campos, Bhaskar Mitra, Emine Yilmaz, and Bodo Billerbeck. Orcas: 18 million clicked query-documest pairs for analyzing search. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, 2020. 1, 1, 10
Lisa Dunlap, Yuhui Zhang, Xiaohan Wang, Ruiqi Zhong, Trevor Darrell, Jacob Steinhardt, Joseph E. Gonzalez, and Serena Yeung-Levy. Describing differences in image sets with natural language. In Proceedings of the IEEE/CVF Conference Computer Vision and Pultern Recognition (CVPR), 2024. 4, 25
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars, 2025. URL https:/ //tpss///axvv..o/ org/ abs 22003 .11.3..1.33 313.
abos
Jianfeng Gao, Chenyan Xiong. Paul Bennett, and Nick Craswell, Neural approaches to conversational information retrieval. Springer, 2023. 10
Gemini. Grounding with google search, 2024. 1,9
Gemini.
Gemini 2.5:
Our most intelligent ai model, 2025. gow.loggggggleeehchhogooo Mochnnoooyy/ooooo www.loggggggoloooooooooooooooooo google deepmind/ mhhh
URL
8
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models, 2024. URL https: : / /awxiiivvv//./////////.aa///avva org/ abs 2407.21782
Maarten Grootendorst. Bertopic: Neural topic modeling with a class-based tf-idf procedure, 2022.
URL htttttttt/aaiiiiii org. absp/bbs//2233 05794. 22
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. Realm: Retrieval- augmented language model pre-training, 2020. URL https: /tt./aaivvvoo.//. wwwp:/wwwww/a////////oo/// 08909. 9
Rujun Han, Yuhao Zhang, Peng Qi, Yumo Xu, Jenyuan Wang, Lan Liu, William Yang Wang, Bonan Min, and Vittorio Castelli. Rag-qa arena: Evaluating domain robustness for long-form retrieval augmented question answering, 2024. URL https::tt////tttt /atpss////aiiv..// org/
wwgg/bb////ee abs//b///////// abw/aab//15 /2707 13998. 9
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021a. URL https://
arxiv. wwt.:oo//abbb/00//o 20t.b///2..0.3300.... 03300. 9
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021b. URL ht /ttt:///aiiivii wtt./oo////////////// 03874,
12

---

Published as a conference paper at ICLR 2026
Jacob Hilton, R Nakano, S Balaji, and John Schulman. Webgpt: Improving the factual accuracy of language models through web browsing. OpenAl Blog, December, 2021. 1, 9, 10
Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padman- abhan, Giuseppe Ottaviano, and Linjun Yang. Embedding- -badd retrieval in facebook search. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2020. URL http: //xx doi. wtt.oogg//oooooo 10. /141/33944422 3403305. 340333 9
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024. URL https: : /arxivv.vvv. org/ /abs//////// /2403.
07974. 9
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL
abs///as///// 2310. 06770. 9
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017. URL httpppp://axxx.orrg abs/1705. 03551.
Gregory Kamradt. LImtest needleimahaystack, 2023. URL httttt http://///gggtttt..oo/o///aatttt maammmiinnn RREAIE/EEAAAEE.EE. md.
SLMTest_Neediae htw.e/eeeeeeeaaaaaattaaaaaa/a..a http://attaaaaa.aooooaa///////aa
Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, San- ket Vaibhav Mehta, Lalit K. Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q. Tran, Quoe V. Le, and Orhan Firat. Big-bench extra hard, 2025. URL https: //tp:///aavvv//eee/eeeeeeeaaaeea wwweiiiorrr//.o abs 19187. 9
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2023. URL 11916. 9
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmennte generation for //arxiv. //arxiv.org/abs//............. hbtp:/bb/2200 2010111111.........111111111110.
intensive nlp tasks, 2021. URL https:
Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jingyuan Wang, Jian-Yun Nie, and Ji-Rong Wen. The web can be your oyster for improving language models. In Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguisties. 2023a. URL
https: http///aaaaaahlaaoo
org/ 2023. El Findings-act 46/.
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for aoll-nueeeted 1lms, 2023b. URL
https: arxiv.. org/ wtwp/abs/////ooooooooo/o//////a/ / 2304 08244. 9
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data high-quality benchmarks: Arena-hard and benchbuilder pipeline, 2024. URL https: / /aaaiiiiiiv.///////aaaaa/aavavvv org/ abs / /2004 11.406.11999 11939. 9, 35
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking LLMs with challenging tasks from real users in the wild. In The Thirteenth International Conference on Learning Representations, 2025. URL https: / //tp//pppeeeeeeeeo///e wttp:/forummid-MEcCC..
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report, 2024a. URL
attps://arxxx org/ abs 19437. 9
Junhua Liu, Yong Keat Tan, Bin Fu, and Kwan Hui Lim. Intent-aware dialogue generation and multi-task contrastive learning for multi-tum intent classification, 2024b. URL https: : / /arxivv. org/abs/ 2411 14252.
13

---

Published as a conference paper at ICLR 2026
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng. Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agenthench: Evaluating llms as agents, 2023. URL http:pps//aaxxi.org//aaaeeeaeeee 03688.
Fengran Mo, Kelong Mao, Ziliang Zhao, Hongjin Qian, Haonan Chen, Yiruo Cheng, Xiaoxi Li, Yutao Zhu, Zhicheng Dou, and Jian-Yun Nie. A survey of conversational search. ACM Trans. Inf. Syst, 2025. doi: 10.1 10/1745377 URL https: //ois http://ooooorg///////// 1tte////..11//5559111
10
Solmaz Seyed Monir, Irene Lau, Shubing Yang, and Dongfang Zhao. Vectorsearch: Enhancing document retrieval with semantic embeddings and optimized search, 2024. URL https:// arxiv. org/ abs 2409 11209.173800 17383.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark, 2023. URL https::///// /arxivv org/ abs abs/abs2200 22100220... 07316.. 9
OpenAl. Introducing chatgpt search, 2024.
URL https:// http://pppaaa..ccoo com/ index/ httpps//opeeai.ccom
serroduuiing-chatppt--- search/. 1,
OpenAl. Introducing gpt-4.1 in the api. 2025. URL https: https://openai.coom com.///ei..cccm////xx////// cww.indew////oo/coooooccccccc.ce
9pt-4-1/. 20
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022a. URL https: /ttp///aavvvvvvvv///a//eeeeaaava org/ abs 2203.2033.2 02155. 9
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Chris- tiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems. Curran Associates, Inc., neurips cc/paper fareerr__illleeeeeeeeeeeeeeeeeee fileff filesspapez/2022 Paper- Conference pdf. pdf. 10
2022b. URL https: neureee.pdddnnneneeee.eeee.
Lawrence Page. Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web, 1998. 9
Shishir Patil, Tianjun Zhang, Xin Wang, and Joseph Gonzalez. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems, 2024.
Perplexity. Sonar by perplexity. httt:: docs pewplexity. 1,
Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. Okapi at trec-3. In Proceedings of the Third Text REtrieval Conference (TREC-3), 1994. 9
Timo Schick, Jane Dwivedi-YV Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023. URL httph:tp..... //tp:///aiiivve
http:://wwwww.../aa//////sss http://///////////eabb 04761. 9, 10
Chirag Shah, Ryen W. White, Reid Andersen, Georg Buscher, and Scott Counts. Using large language models to generate, validate, and apply user intent taxonomies, 2023. URL https:
// //arxivvorr http://ogggb/bb//aa/o/oo/aa 2209 13063.
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. URL
https: http://////xxvv.x//////// /303 313661.....1.
Mark Steyvers, Heliodoro Tejeda Lemus, Aakriti Kumar, Catarina Belém, Sheer Karny, Xinyue Hu, Lukas Mayer, and Padhraic Smyth. The calibration gap between model and human confidence in large language models, 2024. URL https: / /api. semant inanniicsen icshhhlarr.crrr org/CorpusID: 267211 5
14

---

Published as a conference paper at ICLR 2026
Kelly Tang, Wei-Lin Chiang, and Anastasios N. Angelopoulos. Arena explorer: A topie modeling pipeline for IIm evals & analytics, 2025. 22
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 2), 2021. URL https: opw.review.eeeeeeeepeepe. net forum? diw.mmdTd-dwwwwx jeJ. 9
Wei-Lin Chiang Tianle Li, Anastasios Angelopoulos. Does style matter? disentangling style and substance in chatbot arena, 2024. URL https::/bbbbt /bllg. Imarena www.aeaaaaabblog.....a ai/blog/2024
coneeeoooooo////ooooo.o 2, 5,5
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL https: : aawii.iiiivv///i/ org/ absa wwtp//02..........ee.......e.... 1307.13971. 9
Ellen M. Voorhees and Donna K. Harman (eds.). TREC: Experiment and Evaluation in Information Retrieval. MIT Press, 2005. 9
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023. URL https: arxiv. org/ abs 220..22222... 11171. 9
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning large language models, 2023.
URL https:://aattt http://axivv.o// org/ abs/2201. 11903. 9
Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality large language models, 1, 2, 3, 4, 9, 35, 36
2024. URL https: / wwxiv.ooooa........ aab /1////411..033
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese Browsecomp: A simple yet challenging benchmark for browsing agents, 2025. URL
12516. 1, 2. 3, 4, 9, 36
Minghao Wu and Alham Fikri Aji. Style over substance: Evaluation biases for large language models. In Proceedings of the 31st International Conference on Computational Linguistics. org/
Association for Computational Linguisties, 2025. URL https: sectps://aclantholllo
2025. co mowinggmainn........oo......aaam
Tsung-Har -aang-Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E. Gonzalez, Trevor Darrell, and David Chan. Visual haystacks: A vision-centric rredle-in-a-haystack benchmark. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
//openreview.oo net. 1ormmiid=-JCNPPLLE..
Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica. Rethinking benchmark and contamination for language models with rephrased samples, 2023. URL https:
//waxiv. abs 2211 04850.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018. URL https::/ http:////aaa/hhhhhooo org/D18 1259/.9 .9
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L.. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023a.
URL https::/ /atpsv/////iv.va/ org/ abs/2305... 10601.9 9
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023b. URL https: //arxiv.
org//bs///// 2210 93 10
15

---

Published as a conference paper at ICLR 2026
Zhongshen Zeng, Yinhong Liu, Yingjia Wan, Jingyao Li, Pengguang Chen, Jianbo Dai, Yuxuan Yao, Rongwu Xu, Zehan Qi, Wanru Zhao, Linling Shen, Jianqiao Lu, Haochen Tan, Yukang Chen, Hao Zhang, Zhan Shi, Bailin Wang, Zhijiang Guo, and Jiaya Jia. Mr-ben: A meta-reasoning benchmark for evaluating system-2 thinking in llms, 2024. URL htw.t:ttt///ccm hatpssa//aaciii.org/aa.s. 2406. 13975. 9
Jiajie Zhang, Yushi Bai, Xin Lv. Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, and Juanzi Li. Longcite: Enabling LLMs to generate fine-grained citations in long-context QA, 2024. URL https: / /opnreveevvv ew net/ / forum? d=mMXdHyBcHh,
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: Im chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024. URL https:
www.:eeee//oouumoeee wttpat//eoooommmmmmaaaaaaa11211 1,
9,37
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing. Joseph E. Gonzalez. Ion Stoica, and Hao Zhang. Lmsys- Ahat-Im: A large-scale real-world IIm conversation dataset, 2024. URL https: ptttcps://axxv..oo/ attaaaaxvvvoooo//aaa abs/2309. 11.1199999 1999
Ruiqi Zhong, Charlie Snell, Dan Klein, and Jacob Steinhardt. Describing differences between text distributions with natural language. In Proceedings of the 39th International Conference Machine Learning. PMLR, 2022. URL https::/proceeeee zhong22a. html. 4, 25
mlr www..m.r....veeee.mmm
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024. URL https: : //arxiiv.orggg org/abs/ 2307. 13854.
16

---

Published as a conference paper at ICLR 2026
APPENDIX
Appendix A describes platform design, supported model, and policies of our data collection process. Appendix B outlines intent classification, topic modeling, and filtering procedures.
Appendix C includes details (e.g., parameters, prompts) on the dataset differencing experiments. Appendix D includes model performance and user preference analysis on Search Arena.
Appendix E outlines citation analysis, user preference experiments, and citation attribution pipeline. Appendix F includes additional results on the coss-benchhark analysis.
Appendix G includes comparative analysis with additional related datasets.
A SEARCH ARENA PLATFORM
(aatal
Laudetbeard
Pronge Landebaird
X Chatbot Arena (formerly LMSYS): Free Al Chat Compare & Test Best Al Chatbots
Disond Uwitee H1 [oindd] (atue
Competiion
How it Works
aloel sysppanten enepyeen clutionn Soastien Javenn ceninan))
Choose thei vascas
Play naietlit devetyremely your wontounnt count.
lmag
dregaon gens-inngo
Chatbot Arena LLM Leaderboard
cammurtty
Chat now!
Eosedits ae rectstirn et mode
Canditiono Most cleaty 301
Dylyy Femroet
Whate the westher Derkelyy Inday!
Monsday, 221 Lowesar carc, High sir (ond, Dounition thauey cool
with band lasm cinads
Tuentlyy low srsp 1nq, right sere new rittion dluuds
in Thehegtoemmeeer
maminn biloued partyy
sa'v
cond, desoriation
of cloud,
mph Thend approd chunceafien oliann, nurannmmedd mendyin thereening a
aouthwest
thunsley, May Lowe burck zonh tand, Deeolprion tfe ount with
ereus of doads, thenour
meeinn mearing siets thepeeptt ppeepptatooncoedd
Saturday, May 17 Lose AP1 prc,, Highoor oft (ard Deac plore Low douch
laper of eltamoon, shiing toiroo soingnwing cand toro B
fellowed by poms sun
May Lowetshh (ond) Hight torkt, Desorittion taw
theckyasccnntion imprrvs temperttons medeette
moke for ppratrpgnngty messeay.
Survise ou 60 swdee
Ciartinn
Citation
"Br bettter
bettae
Tic
Both are bad
RepoChat
Tediimage
Send
Figure A.l: Search Arena User Interface. Search Arena is integrated into the Chatbot Arena ecosystem, sharing the same front-end design and entry point. As shown at the bottom, it is implemented as a separate tab. The central panel displays side-by-side chat interface with two anonymous models, each providing responses that include clickable inline citations and expandable reference links. Users can engage in multi-turn conversations and cast vote at any time during the conversation using the four feedback options (A is better, B is better, Tie, and Both are bad).
As described in Section 2, Search Arena (https: arena.eeaaa.. ai /search) is open, crowd- sourced evaluation platform for search-augmented LLMs, launched on March 18, 2025. It should be noted that the platform's 's user interface has been changed and the data for this study has been collected through the legacy interface (see Figure A.1). This section outlines our data collection and release protocols, followed description of the supported models and key design decisions.
17

---

Published as a conference paper at ICLR 2026
A.1 DATA
Data Collection. The Search Arena platform does not require user login, but users must explicitly accept the Terms of Service before using the platform. Once accepted, users access dedicated tab within Chatbot Arena to interact with search-enabled models. This setup naturally leads to more information-seekle seeking queries compared to the default Text Arena interface. For each prompt, responses from two anonymous models are displayed side-by-side, and users may cast a preference vote at any time during the interaction. The user interface is shown in Figure A.I. The full text of Terms of Service is at Figure A.2.
Over the course of a 7-week data collection period, the platform recorded an average of roughly 800-1,500 conversations per day. After filtering examples with server errors, inconsistent configurations, or other quality issues, we retain approximately 24,000 conversations, about half of which include human preference votes. Figure A.3 shows daily traffic trends, with spikes aligned to major model updates or platform announcements, and dips weekends.
Terms of Service
Users are required to agree to the following terms before using the service: The service is research preview. It only provides limited safery measures and may generate offensive content, It must not be used for any illegal, harmful, violent, racist, sexual purposes. Please do not upload any private information. The service collects user dialogue data, including both text and images, and reserves the right distribute it under Creative Commons Attribution (CC-BY) may only use this website for your personal or internal business purposes. You must not access the website programmatically scrape extract data, manipulate any leaderboard or ranking, authorize or pay others to access or use the website on your behalf. Unauthorized use may result in suspension or termination of your access, including access by your organization.
similar license. You
Figure A.2: Term of Service of the Search Arena Platform.
Fittered Traffic
Traffic with Votes
800
700
800
#Convs
500
400
300
200
100
Mar
2025
Mar
Mar
Apr
Apr 13
Apr 20
Agr 27
May 11
Date
Figure A.3: Daily traffic on the Search Arena platform.
Data Release Policy. Per our Terms of Service (Figure A.2), all conversation data is released under a Creative Commons Attribution (CC-BY) license. To protect user privacy, we apply automated de-identification using Google's 's Data Loss Prevention (DLP) API, which removes personal identifiers such as email addresses, credit card numbers, and API keys. Approximately 2% of examples were flagged by this process. Users may also contact the platform provider to request their records be removed from the released dataset through privacy arena.ai. Details on the privacy policy of the platform provided here.
18

---

Published as a conference paper at ICLR 2026
Table A.1: List of supported models and their configurations on Search Arena. We evaluate OpenAT's web search API, which differs from the search feature in the ChatGPT product.
Provider
Model
Base Model
Details
pplx-sonar
sonar
conlig
ypix-sonar-Pp
sonar-prr
Default config
Perplexity
golx-sonaa-pro-high
sonar-pro
seazch sxt__eeeeegggg
pplx reasoning
reaar-reusnning
Default config
wow.soonn--ssnnnn--oeeee prooohiggg
sonar-rcasoning-pro
seazch_com cwtcccc_oooottteeeeeee ext__set___te sizeehigh
gemini 2.0. 2.0-flshh grounding
gemini- 2.0. flash gemini-2.5-faas
With Google Search enabled
Gemini
gemini-2.5 gemini-2.5-fash--neee 5-Baash-rruunding
Google Search
gemini 2.5 pro grounding
2.5 pro exp-03 0..--33.2
With Google Seurch emabled
upè gpt-to-mini-- search preview upi gpt 4p search preview
gpt-4o-miini
config
OpenAl'
gpt 4o
Defauli conlig
api-gpt-4o-seme sapt-4o-seareh-pe search-previewwhig
gpt-4o
search_comm extt s ze user location Teature enabled
api ypt 4o search preview high loc
gpt 4o
A.2 MODELS
Search Arena currently supports 12 search- auggened LLMss from Perplexity, Gemini, and OpenAI, as summarized in Table A.1. Unless otherwise noted, we group models with different inline cita- tion styles (elaborated below) of the same base model. Each model is accessed via its provider's 's public API using the default configuration, including search_ search_context, www.eottttttiioeeeeeeeeeeeeeeeee sizmeedd um for Per- plexity and OpenAl models. We also evaluate several additional variants, including (1) OpenAl's search-previer
gpt 4o-oooss search preview with user_locats ion enabled and (2) Perplexity and OpenAl models configured with search_context_sinn siieeeeittiieeeeiihheeee Figure D.1 shows the number of battles model.
Model Anonymity and Citation Style Control. Each provider uses distinct citation style, which may unintentionally reveal model identity. At the same time, citation formatting can influence user preferences. To manage this tradeoff, we implement citation style randomization mechanism: model outputs are rendered using either a standardized format or the provider's original style. This approach reduces the risk of user-side de- de-anonyymzztion while allowing user behavior. We found that the inline citation style does not significantly affect user preferences and model rankings.
study how citation style affects
B
DATA STATISTICS
B.1 LINGUISTIC FEATURES
Users' demographic information in Search Arena, based on country codes extracted from IP addresses, is shown in Figure B.I. The dataset includes 1 11,650 unique users across 136 countries, with the United States (18.7%), Russia (9.9%), Germany (5.9)), and China (5.5%) the top four.
Language distribution (of the top 50 languages) of Search Arena prompts is shown in Figure B.5. The prompts span 71 languages with English (56.4%) Russian (11 11.4,) and Chinese (6.8%) as the top three.
As shown in Figure B.2, Search Arena prompt lengths vary across intent categories. As expected, Text Processing (85.1 words) and Analysis (72.7 words) prompts are generally longer, while Explanation (24.7 words) and Factual Lookup (16.3 words) prompts shorter.
Search Arena conversations multi-turn, with 22.4% of all conversations containing more than one turn. Specifically, there are 3,288 conversations with turns, 966 with 3 turns, and 460 with 4 turns. The distribution is shown in Figure B.3.
B.2 USER INTENT ANNOTATION
In Subsection 2.2 and Figure 1, we briefly introduced the intent annotation pipeline and its high-level findings. Here, we provide additional details of the pipeline.
First, the three co-authors of the paper did open-text annotations 100 English prompts randomly sampled from the collected dataset. The annotators then met to consolidate their annotations into a taxonomy of primary and secondary user intent categories, including definitions and representative
19

---

Published as a conference paper at ICLR 2026
examples. The final taxonomy includes the following nine intent categories: Factual Lookup. Information Synthesis, Analysis, Recommendation,, Explanation, Creative Generation, Guidance, Text Processing, and Other. Full category descriptions are shown in Table B.1.
To validate the taxonomy, the annotators labeled subset of 100 prompts. The human inter-annotator agreement, measured by Cohen's 's Kappa, ranged from 0.65 for primary labels to 0.79 when consider- ing top-two matches (substantial agreement).
We then scaled the annotation to the full dataset using GPT-4.1 (OpenAI, 2025), using the same in-context examples from Table B. The total annotation cost was approximately $20 USD. To evaluate label quality, we compared GPT-4.1 annotations with human labels on 150 samples drawn from the top three languages our dataset (English, Russian, and Chinese). The resulting Cohen's Kappa score of 0.812 on top-2 intents indicates strong agreement. The full GPT-4.1 prompt is provided below.
Prompts for LLM-based Intent Classification
You are an impartial classifier.
TASK 1:
Primary intent: choose one category that best matches the user's intent, always ask yourself, what is the user trying to get the model do with this query. Considering them fol following ng the categories one by one in order.
TASK 2: If there is clear secondary goal, choose one additional intent label that is also present in the query. Otherwise, return "Unassigned" if no clear second goal exists.
Allowed intent categories:
1. Text Processing
2. Creative Generation
3. Factual Lookup
4. Info Synthesis
5. Analysis
6. Recommendation
7. Explanation
8. Guidance
9. Other
Here are a few examples that you should follow, try to generalize boyond these examples but hold true to their semantic moaning:
(FEW SHOT IN-CONTEXT EXAMPLES)
Respond only in valid JSON, only choosing exact.ly one category to fill the values:
parinary_intent": intene": "<primary intent label>* "secondary_inte intent"; "<secondary intent label>*, "ressoning"; "cyour zeascning classification''
User query: (USER PROMPT]
20

---

Published as a conference paper at ICLR 2026
Table B.I: User intent taxonomy including category descriptions and examples.
Category
Description
Examples
Text Processing
Users request linguistic tasks such summarizing, translating, paraphrasing, refining text.
Summarize <website>.
Translate this paragraph from Span- ish to English.
Rephrase this email professionally for an investment -aanking applica- tion.
Users request original creative outputs,
Creative Genera- tion
including fictional scenarios, satire, po- etry, storytelling, or other forms of artis-
Compose witty review of ITV's Big Brother.
Create a humorous poem about SpongeBob.
tic language generation.
Write short fantasy story set on Mars.
Users seek to retrieve precise, ob- jective fact or specific information (what/who/when-iik questions).
Factual Lookup
Who invented lambda caleulus?
How many planets are in our Solar System?
When was the Mona Lisa painted?
Users seek a concise, aggregated sum- mary that combines facts or perspec-
Info. Synthesis
List the key functions of the U.S. Congress.
tives from multiple sources. The focus
Summarize the main events the French Revolution.
is integrating factual content without
requiring reasoning, subjective interpre- tation, personalization.
Users seek a reasoned judgment or breakdown of topic, often involving comparison, evaluation, weighing dif-
Analysis
Analyze the pros and cons of nu- clear energy.
When will we reach artificial gen- eral intelligence?
ferent perspectives, and syntheszing ma-
terial from many sources. Often open- ended.
What were the strategic implica- tions of the Cuban Missile Crisis?
Users request suggestions advice tai-
Recommendation
lored to particular constraints, prefer- ences, specified criteria. Often im-
Best laptop for machine learning under $1500.
like literary fiction what books should I read?
plies personalization or ranking.
What programming language should I learn for web develop- ment?
Users seek detailed clarifications, educa-
Explanation
tional insights, or thorough claborations aimed at better understanding cept, process, or phenomeneon. May
What led to the fall of the Roman Empire?
Can you explain the Heisenberg un- certainty principle?
ask "why"" or "how" something works without implying action.
How does gradient-descent opti- mization work in machine learn- ing?
Users request instructions, procedures, or practical advice intended to accom-
Guidance
How do install Docker Ubuntu?
plish specific tasks, typically involv-
ing sequential steps or troubleshooting.
How do renew my passport on- line?
Usually framed "how to" or involving action.
How do replace the battery in a MacBook Air?
21

---

Published as a conference paper at ICLR 2026
Category
Description
Examples
Prompts that don't t fit neatly into any of the categories above. Prompts may be malformed, declarative in nature, or lacking in meaningful user intent.
Other
pizza.
The table hit my head.
Talk about something please.
Coome
Figure B.1: Search Arena Users' Demographics. Search Arena data includes 11,650 unique users across 136 countries.
1000
Log(nurn words)
10
Taxt Procetung
Anslysks
Creative Generation
Guidlance
Info Synthepis
Recammendttioon
Explanation
Factual Lookup
Figure B.2: Search Arena Prompt Length Distribution by Intent. Prompt lengths vary across intent categories. Text Processing (85.1 words) and Analysis (72.7 words) prompts generally longer, while Explanation (24.7 words) and Factual Lookup (16.3 words) prompts are shorter.
B.3 TOPIC MODELING
To analyze topic diversity across benchmarks, we apply the BERTopic framework (Grootendorst, 2022) to the 11.664 unique English prompts in the Search Arena dataset. We generate prompt em- beddings using OpenAl's text- eext--mbbeldig----aarge. model, perform dimensionality reduction and clustering via UMAP and HDBSCAN, and summarize each resulting cluster using GPT-4o (Tang et al., 2025). We adapt the Arena Explorer (Tang et al., 2025) methodology for the Search Arena dataset. Figure B.4 illustrates the distribution of topics derived from BERTopic clustering
22

---

Published as a conference paper at ICLR 2026
the Search Arena prompts. The prominence of categories such as Technology Comparisons, Market Analysis, and Entertainment Characters, alongside a long-tail of niche domains, highlights the dataset's breadth and its suitability for evaluating search-augmente LLMs under diverse topics.
77.6%
10k
Count (log scale)
13.7%
1K 4,0%
1.9%
1.3%
1.0%
0.6%
6++
Turns Conversation
Figure B.3: Search Arena Conversation Length (Number of Turns) Distribution. Search Arena chats are multi-tury with 22.4% of conversations containing more than I turn.
Search Arena Topics by Proportion
25%
22.0%
Proportion
13%
10.6%
10%
6.2%
6.2% 5.7% 5.3%
5%
3.5%
3.5% 3.1% 3.1%
2.6% 2.6% 2.2% 1.8% Famous Pigrures
Techmology Comparisons
Entertalnment Characters
Market Analyois
Currant Evonts
Information Updates
Al Models
Divorse Subjects
Satirical Critique
Historical Analysis
Sclence Discussions
Model Deploymont
Health
Gaming Strategios
Consumer Shopping
Updates
Recommendation
Figure B.4: Top Topic Categories in Search Arena. This figure shows the distribution topic clusters. The most prevalent topics include Technology Comparisons (22.0%), Market Analysis (12.3%), and Entertainment Charucters (10.6%). Less frequent but still diverse topics include health and shopping. The long-tail tail distribution reflects the breadth of real-world usage of search-an sarch-auummnted LLMs.
23

---

Published as a conference paper at ICLR 2026
Engish
56.428%
Russion
11.380%
German
3.047%
Spanish
2.731%
Japanese
2.689%
Vietnameee
2.573%
Korean
2.527%
2.473%
Itallan
3.322%%
Indonesion
0.811%%
0.715%
Porsion
0.586%
Turkich
0.495%
0.411%
Ukannan
0.303%
That
0.291%
Catalon
0.237%
Dunch
0.212%
0.204%
Swedish
0.158%
Hebrew
0.067%
Bugarinn
Finrish
0.058%
Serbian
0.050%
0.046%
Norwagian
0.048%
Esporanto
Slovenion
0.033%
Danish
0.033%
0.033%
Slovak
Ga Idaa
0.025%
Crcation
0.025%
Amonion
0.021%
Tibttan
0.017%
Uthuanian
0.017%
Macodonian
0.017%
Khmor
0.012%
Maley
0.012%
Nopah
0.012%
0.012%
Goorgian
0.012%
Besqua
0.012%
Tegelog
Uzbak
0.008%
100
1000
10000
Count (log scale)
Figure B.5: Search Arena Language Distribution (top 50). Search Arena prompts span 71 languages.
24

---

Published as a conference paper at ICLR 2026
C DESCRIBING DIFFERENCES
To extract interpretable differences across text corpora (e.g., prompts, responses), LLM- based dataset differencing methods (Zhong et al., 2022; Dunlap et al., 2024).
C.I PROMPT DIFFERENCES
To describe differences prompt distributions between Search Arena and other datasets (see analysis in Subsection 2.2), we use 1PT-4.1 to propose properties (hypotheses) across 16 rounds, with 32 samples per group in each round. We then use o3 to filter the top five properties and GPPT-4.1miie to re-rank them. The prompts are shown below. We use the default temperature parameter of 1.0.
We extract the following properties (p < 0.001) for each dataset pair and report the proportion of validation set examples (100 samples) satisfying each property.
Search Arena SimpleQA:
Requests in-depth explanations, analyses, or step-by-ste stes guidance rather than single factual answers (28% vs 0%).
Requests recent real-time information, including current events, product features, or online research (35% 7%).
Seeks technical help, troubleshooting, or comparative evaluations for software, programming. digital tools (17% vs 1%).
Asks for creative content generation, rewriting, or stylistic transformations (stories, poems, satirical pieces, etc.) (13% vs 0%).
SimpleQA A vs Search Arena:
Requests exact factual details such as specific names, dates, numbers, or titles (97% vs 41 41%).
Expects a single objective, verifiable answer rather than explanations or analysis (97% vs 43%).
Avoids subjective, open-ended, or creative requests, limiting queries to factual retrieval (99%
52%).
Focuses on niche or lesser-known historical, scientific, or cultural topies (51% vs 7%).
Presents concise, narrowly scoped questions with minimal background information (91 vs 56%).
Search Arena BrowseComp:
User messages are brief and provide minimal contextual detail (80% 23%)..
Responses expected are immediate functional outputs such as lists, summaries, or code snippets (50% vs 20%).
Prompts seek practical advice or step-by-step instructions for real-world tasks (29% vs 1%).
Queries focus on well-known, mainstream topics, products, or services (75% vs 48%).
BrowseComp vs Search Arena:
Frames the query deductive puzzle investigative challenge (80% vs 3%).
Provides explicit temporal geographic constraints that narrow the solution space (99% vs 25%).
Requires synthesizing multiple detailed clues from different sources to deduce the answer (92% vs 36%).
Search Arena Text Arena:
Requests for current factual information about real-world entities, events, or products (61 25%).
25

---

Published as a conference paper at ICLR 2026
Requests for technical comparisons, evaluations, purchasing guidance devices, soft- ware, or services (16% vs 4%).
Text Arena vs Search Arena:
Seeks analytical, step-by-step solutions to mathematical, logical, or technical puzzles and explanations (21% 4%).
Requests programming assistance, such as debugging code, generating scripts, or explaining programming concepts (23% 8%).
Requests creative writing outputs, including stories, poems, jokes, or fictional scenarios (20% vs 5%).
LLM Prompt for Proposing Distinguishing Properties between Prompt Sets
'The following are separate lists of prompts that users have asked chatbot:
<STAR>>
(text]
<END>
I machine learning researcher trying to figure out the major differences between these two groups of prempts. This is very small portion of the data, so I want the differences to be general.
Please provide a list of the top three differences (separated by bullet points *.") between the prompts in Group A and Group B.
Follow the detailed instructions below:
Icentify properties that are common in Group prompts compared to Group prompts.
The property descriptions should be simple and detailed. Do not produce vague and generic descriptions.
Start the description of each property with "User". Do not keywords like "group A", "group 8", "more common", "less common", "Erequently", "occasionally" your respense.
Do not combine multiple properties / features in a single bullet point.
An example of the desired output format: "Property "Property 2"
"Property 3"
Please order your response in terms of the most common differences between the two groups. Your response:
LLM Prompt for Reducing Proposed Properties
PROPERTIES:
(properties]
Above is list of properties, several of which are similar.
Please reduce this to list of the five most common distinct properties, ordered by frequency of occurrence. list.
There should be no similar / overlapping properties in the final
group multiple unrelated properties features into single property.
Do not use keywords, like "frequent", "rare", "common", "uncommon", etc.
Only respond with a numbered 1ist of properties, other text.
26

---

Published as a conference paper at ICLR 2026
LLM Prompt for Ranking Properties
PROMPT:
<STAR">>
(text)
<END>>
PROPERTY:
<START>
(hypothesis)
<END>
Check whether the PROMPT satisfies the PROPERTY. Respond with Yes If you are unsure, respond with No. Output:
C.2 RESPONSE DIFFERENCES
To describe differences in responses between search and non-search models in cross-arena deploy- ments (see analysis in Subsection 3.3), use GPT-4. to propose properties (hypotheses) across 32 rounds, with 8 samples per group in each round. We then use o3 to filter the top five properties and GPT-4.1-ming to re-rank them. The prompts are shown below (for property reduction, we re-use the same prompt from Subsection C.I). We use the default temperature parameter of 1.0.
For each dataset pair, we extract the top properties and report the proportion of validation set examples (100 samples) satisfying the properties, along with corresponding significance levels (p-values).
Text Arena Setting
Search non-search model responses to Factual Lookup prompts:
Includes precise quantitative or technical specifics such exact dates, numerical values, specifications, or code snippets (53.8% vs 41 1)) (p = 0.26).
Search vs non-search model responses to Info Synthesis prompts:
Incorporates precise data, statistics, dates, names, and domain-specife terminology (80.1 % 61.9%) (p = 0.18).
Non-search vs search model responses to Text Processing prompts:
Uses explicitly structured formatting (numbered or bulleted lists, headings) to organize information (74.4% 56.4%) (.10.
Search Arena Setting
Search non-search model responses to Factual Lookup prompts:
Provides extensive background context or explanatory preamble before addressing the specific question (83.3% vs 72.2%) (p 0.26).
Search non-search model responses to Info Synthesis prompts:
Explicitly states limitations, uncertainties, or caveats about the information provided (71.7% vs 52.2%) (p = 0.05).
27

---

Published as a conference paper at ICLR 2026
LLM Prompt for Proposing Distinguishing Properties between Model A and Model B Responses
The following is list of Model and Model outputs set of user questions:
<START>
[text]
<END>
machine learning researcher trying to figure out the major differences between the outputs of model and model В. This is very small portion of the data, so I want the differences to be general.
Please provide a list of three top di fifrrrnccss (separated by bullet points "*") between the outputs of model and model B.
Follow the detailed instructions below:
Icentify properties that are more common in model outputs compared to model B outputs.
The property descriptions shculd be simple and detailed. Do not produce vague and generic descriptions.
keywords like "model A", "model B", "more common", "less common", "frequently", "occasionally" in your response.
Do not combine multiple properties / features in a single bullet point.
example of the desired output format: "property "property 2"
"Property 3"
Please order your response in terms of the most common di differences between the two groups. Your response:
LLM Prompt for Ranking Properties
QUESTION:
<STAR>>
(question)
<END>
ANSWER:
<START>>
(answer)
<END>
PROPERTY:
<START>
(hypothesis)
<END>
Check whether the ANSWER satisfies the PROPERTY. Respond with Yes or No. If you are unsure, respond with No. Output:
D LEADERBOARD AND GENERAL FEATURE ANALYSIS
D.1 LEADERBOARD ANALYSIS
Models supported on the Search Arena platform are shown in Table A.1. Number of battles across models is shown in Figure D.I; pairwise battle count is shown in Figure D.2.
Average win rates across models are shown in Figure D.3. Pairwise win rates are shown Figure D.4.
28

---

Published as a conference paper at ICLR 2026
2500
2510
2510
2485
2353
2257
2238
2065
2000
Number of Battles
1822
1500
1041
1000
0
sanar reasonning propppo--io
sonar- cilee-pro-tiee -oro--tggh
got wew----arh-hhhhheeeeee h-ght
sonar ressoning
A---sarch--hig
gemini-2 Somin--..5-pra pro-grounding
gpt-4o seaeehc
gemini 2.0 fo--tash-graaeee groundinn
gPt ho--4o-inn-ssne
sonar
gemini-2 s-ffash-graunding graundingnddng
sonar-pr pro
Figure D.I: Battle Count Distribution. Number of battles across Search Arena models. The distribution is not even due to (1) different sampling weights and (2) models were not added to the platforms at the same time.
Model
stmiiaa 2. aerorrn
gamiis-2 D dest prounding gummin-2 5 S-llet- groundon
comerraiin proaa high
Sat dhr saarch high
gat-Aee-a ste-saacchh search
gatt de mote miti--siinn stiroo
preate high
mee Wro
soter
cora-resscnngg 1oo
238
266
279
199
102
143
196
216
213
100
199
117
gurres 2.5--pm grounding
266
116
200
175
95
128
85
247
203
201
144
117
228
204
231
157
171
203
sonar reasoning
247
201
247
A.
sonar
123
272
gerti 22.5-aahh grounding
gemtn 220-aaahhhh- grounding
218
24:7
searchhhiii
223
201
202
4o oearch trigh-loc
200
201
243
gpt-4p-search
Figure D.2: Pairwise Battle Count Distribution. Number of battles between Search Arena models.
Consistent with Chatbot Arena's ranking system (Chiang et al., 2024), we use Bradley-Tery regres- sion (Bradley & Terry. 1952) to calculate model coefficients and then re-scale to match the scale of the Elo rating system. The resulting model scores are shown in Figure D.5.
Search Arena model ratings and ranks are shown in Table D.1. We also compute model scores and the leaderboard for two subsets of the full dataset:
English and non- English prompts: The gap between the two top models increases, with sonar-rressoning-poo--bbbe performing better non- Englishh prompts. Addition- and gemini- 5- www.saagrsuudhaae
ally. gemini-2.. w-t.-o0ff-ghhoooooe perform better non-English prompts.
29

---

Published as a conference paper at ICLR 2026
0.66
0.65
0.62
0.57
0.56
0.5
0.46
0.44
0.4
0.38
0.38
Average
0.3
0.2
0.1
sunar remsonig pro high
gemini- S-pro graunding
sonar sonar-pro high
sonar reasoning
gemini-2 9-Ilast--un --laait-grrunding grounding
sonar-pro
sonar
gemini- 2.0 w-wast--paaae.eo B-fbashggrunndng grounding
gemini-2 S-rro
got dordooo-tarc--hhee saarrhh-igh-iie
gBt-4e 1a8-40-search
got-4o-searct-ii high
apt- pae-4a-miti--oe search
Figure D.3: Win Rate Distribution. Average win rates of Search Arena models.
Model
gemini- 2. DD flas go coundiig geriin--2 2. S-flssh gr suuning
2.5 semind
got-4o- search-rrchceo Righ loc
spt 1opt-40-ee search ot hight
spt spt-4p- 1el-----miiiii minimini-en mini----cccchh
soner- recsoning
got Ap-search
pre-hipe high
stoaa wro
sonar
coner-eiscoing uii-o
0.30
0.48
0.35
0.57
0.58
0.58
0.65
0.74
0.74
0.75
0.79
0.88
gemini-2. Simmmii-.--r---rruuddn 0.52
0.50
0.52
0.61
0.64
0.55
0.73
0.73
0.79
0.77
0.76
0.76
sonan-pro-tigh 0.45
0.48
0.50
0.51
0.53
0.53
0.72
0.69
0.76
0.75
0.73
0.75
0.7
sonar-reasoning 0.43
0.39
0.49
0.50
0.48
0.60
0.61
0.60
0.71
0.74
0.72
0.76
soner-pro
0.42
0.36
0.47
0.52
0.50
0.54
0.51
0.63
0.58
0.77
0.09
0.69
0.6
sonar 0.42
0.45
0.47
0.40
0.46
0.50
0.59
0.59
0.64
0.69
0.69
0.74
Model
0.5
gemini 2.5 flash grounding
0.35
0.27
0.28
0.39
0.46
0.41
0.50
0.57
0.67
0.72
0.69
0.74
cemini-20-fllah--nne g-flishh-goonnd
0.26
0.27
0.31
0.40
0.38
0.41
0.43
0.50
0.56
0.57
0.52
0.68
0.4
4o search high
0.26
0.22
0.24
0.29
0.42
0.36
0.33
0.44
0.50
0.42
0.53
0.55
gpt 4o search high-lod
0.25
0.23
0.25
0.25
0.23
0.31
0.28
0.43
0.58
0.50
0.53
0.63
gpt-4o-search
0.21
0.24
0.27
0.28
0.31
0.31
0.31
0.48
0.47
0.47
0.50
0.58
4o mini search
0.12
0.24
0.25
0.24
0.31
0.26
0.26
0.32
0.45
0.38
0.42
0.50
Figure D.4: Pairwise Win Rates. Pairwise win rates between Search Arena models.
Factual and non non-Factur Factual prompts: We observe a clear split in the leaderboard on the Factual subset. Additionally, the gap between the three top models (including sonar-pro-high) "zeros out" on the non-factual subset.
D.2 RESPONSE LENGTH ANALYSIS
Average response length distribution across Search Arena models is shown in Figure D.6 (Left). Reasoning models tend to be verbose except for sona conar-reasontn sonar-pre's version
30

---

Published as a conference paper at ICLR 2026
1151
1150
1130
1113
1100
1096
Rating
1072
1050
1007
1000
999
973
950
geminn--2 pro- pro- proundler pruuding
ronar-reedoning pro-high
Honar-pro-higgh
sonar reattoig
Sonar-pro
sonar
gemini-22 2.5 5-ftast-gaa gr grounding
gemini-2 f-fassh prouddng
gPt-40 3Pt-Ao--earct-hia searchh-high
BPt-4o--earch- high-loc
9Pt-4o spa-ach
SP Poot-4o-mini-ssaal searhh
Figure D.S: Search Arena Leaderboard. Model scores based on Elo-scaled Bradley-Terry coeffi- cients.
Table D.I: Search Arena Leaderboard. Elo-scaled Bradley-Terry ratings along with corresponding ranks (Rating (Rank)). Model ratings based on two subsets of the full dataset: (1) English vs non-English prompts, (2) Factual (Factual Lookup and Info Synthesis) vs con-Factual prompts.
Model
Rating
Rating (English)
Rating (non- moon-English)
Rating (Factual)
Rating (non-Factual)
gemini-2 2.5 ----oorrrroddngg
1150.6 (1)
1146.8 (1)
1158.3
1155.8 (1)
1143.9
sonar-reasoning-pee prooooing--ro
1150.1 (1)
1137.5 (1)
1174.2 (1)
1161. (1)
1142.5 (1)
sonar-pro-ligh
1129.8 (1)
1125.7 (1)
1136.2
1110.1 (3)
1144,1 (I)
sonar reasoning
1113.2 (3)
1112.0 (2)
1117.9 (3)
1105.9 (3)
1121.0 (2)
sonar-pre pro
1099.9
1101.5
1093.1
1092.0
1106.8
sonar
1095.7 (4)
1092.8 (4)
1100.6 (3)
1085.6 (3)
1106.0 (4)
Demini--255-llase flas-groundding
1072.0
1098.4
1059.2 (7)
gemini-22 2. ce-flas--grounding
1035. (8)
1017.8 (8)
1060.7 (5)
1035.5 (8)
1035.4
gpe-to-search-high.
1007.5
1000.9 (8)
1018.7 (8)
1007.7 (8)
1005.6 (8)
gpt-to-scurch-high-loo
999.9
989.9
1019.7
987.1
1009
spt-to-scarch
999.2
999.4 (8)
1002.1 (9)
1000.5 (8)
999.5 (9)
gpt-lo-mini-see
(12)
970.7
973.6
973.1 (9)
969.6 (11)
with higher search context generates longer responses compared the version with a medium context. Response length distribution is uniform across OpenAl's models.
We control for response length difference in the Bradley-Terry model and compute the corresponding coefficient across different subsets of the full data, broken down by intent category (see Figure D.7). For all intent categories, the coefficient (effect) is statistically significant; however, as expected, the effect is smallest for Factual Lookup prompts.
E CITATION ANALYSIS
E.1 CITATION COUNT
Citation count distribution is shown in Figure D.6 (Right). As expected, models with higher search context size cite more sources (e.g., sonar-pro-high vs sonar-pro). Furthermore, reasoning models tend to cite less sources compared to non-reasoning vari- ants (c.g., sonar-reasoning pecc-proohhhhh vs conar-pro-hide sonar). We hypothesize that reasoning models synthesize and filter irrelevant sources be- fore final response generation, resulting in less cited sources in the final response (see Fig- cites fewer sources compared to suggesting that even though both are reasoning models,
sonar-reason
vs
3). Interestingly,
gemini-2, 1ilmii2.555 w.---las--grruudd gemini-2 5-r.5--rr-0e porro-grounding filters out more sources from the final response.
We then control for citation count difference in the Sradley-Terry model and compute the correspond- ing coefficient across different subsets of the dataset broken down by intent category (see Figure E.1). The effect of citation count Guidance (e.g., debugging, problem-solving) prompts not significant. The effect is largest on Analysis prompts.
31

---

Published as a conference paper at ICLR 2026
300
400
Olation Count
100
and sporth Higa
mose coptoronp sosee light
digh
medeniin Ver
Figure D.6: (Left) Response length distribution. Reasoning models and models with high search context size tend to more verbose. (Right) Citation count distribution. As expected, models with higher search context size cite more sources. Reasoning models cite fewer sources compared to non-reasoning variants (e.g., wwwar-reasoneeee neaprarr-raannnn--o-----i- vs sonar-pro-high).
Coeffictnn Estimate
0.5
3.47
0.47
0.40
0.38
0.40
0.2
c.16
Factual Lootoup
Explonation
Recommendestan
Guidancco
Analysis
Creativg Generacion
Processing
Synthesis
Figure D.7: Response Length Control across Intents. Bradley- Terry coefficients corresponding to response length across different intent categories. Length has less effect on Factual Lookup prompts.
0.5
Coofficiont cont Eztimato
0.4
0.35
0.3
0.2
0.1
0.05
-0.1
Guidance
Pretubl Leokup
Info Syrthesis
Tert Procetsing
Recommmnndation
Creative Cenaration
Explimation
Anblycin
Figure E.1: Citation Count Control across Intents. Bradley-Terry coefficients corresponding citation count across different intent categories. Citation count does not have significant effect on Guidance (e.g., debugging, problem-solving) prompts. The effect largest for Analysis queries.
32

---

Published as a conference paper at ICLR 2026
E.2
CITATION SOURCES
To categorize citation sources, we use the following mapping from source domains to categories:
youtube: "youtube.com".
gov_edu: ".gov", ".edu", ".mil".
wiki: "wikipedia", "wikihow", "wikimedia".
us news: "enn.com". "apnews.com", "cnbc.com", "bloomberg.com" "economist.com"
"nytimes.com" "washingtonpost.cmm" "wsj.com". "nbcnews.com" "abenews. ww..soo..mmmm "usatoday.com" "npr.org". "latimes.com", "vox.com". "huffpost.com". "ft.com", "foxnews.come "axios.com", "time.com", "buzzfeed.com" "cbsnews.com" "politico.co". "newsweek.com". "fortune.com: "theatlantic.com" "whattowatch.com" "scripp- com". "investopedia.com "yahoo.com", "breitbart.com' "washingtontimes.com "dailycaller.com", "thefederalist.com", "townhall.com", "pjmedia.com", "westernjour- nal.com" "forbes.com".
foreign news: "reuters.com" "bbc.com", "aljazecra.com". "dw.com", "france24.com com"". "as.com", "elpais.com". "cbc.ca", "theglobeandmail.com "smmh.om..uu, "abc.net.au" "japantimes.com jo. "straitstimes. com", "hindustantimes. com", "thehindu.com", "economic- times indiatimes.comm com"", "iddaanepress.com" "independent.co.uk"" "theguardian.com" "ca- denaser.com", "lemonde.fr"; "vnexpress.net". "ndtv.com".
social media: "tittok.com' "facebook.com" "instagram.com" "x.com", "twitter.com", "linkedin.com com",, "snapchat.com com"", "pinterest.com".
community blog: "reddit.com", "quora.com", "brra.""m,""o"""".. "medium.com", "wordpress.com', "substack.coms "tumblr.com".
tech_ coodheeodigg--iio "github.com". "gitlab.com", "stackexchange.com,,, "microsoft.com", "dev.to"
"codecademy.com" "stackoverfhow.com".
crademic_journat "jstor.org", "springer.com". "sciencedirect.com", "nature.com",
"arxiv.org", "researchgate.com". "biorxiv.org".
retail: "amazon.com", "ebay.com". "walmart.com", "target.com", "bestbuy.com", "costco.com".
other: non-matched domains.
Perplexity
Google
OpenAI
0.4
Proportion Responses
0.3
0.2
0.1
youtube
secial media
community blog
tech coding
academic Jouraa
us news
foreign news
wile
edu
retail
Figure E.2: Citation Source Distribution across Model Families. Models from different providers biased towards different types of sources: (1) Perplexity models prefer citing YouTube, social media, and community blogs, (2) OpenAl models are biased towards mainstream news outlets.
The distribution of citation domain categories is shown in Figure E.2. Perplexity's models tend to cite social media (including YouTube) and community platforms (c.g.. Reddit, Quora). OpenAl's GPT-4o-based models more biased towards mainstream outlets. Google's Gemini models
33

---

Published as a conference paper at ICLR 2026
sonan-pro amaar-po-hhgh
sonar-reasoning pro-high
Proportion of Responses
0.4
0.2
0
youtube
socil menta
community biog
tech coding
acadenic journal
jounal
news
foreign news
wili
gov eduu
retail
Figure E.3: Citation Source Distribution across Non-reasoning (sonar-pro-high) and Reasoning (sonar-reasoning ) Models. Although both systems are based on the same search module, the reasoning model (dark blue) cites more community and blog sources and less Wikipedia articles compared to the non-reasoning variant (light blue).
in between. To further understand the effect of reasoning, analyze the distributional differences of the cited sources by sonar-po per-p-r---- paii--rro-hhhhhh high (non-reasoning) and cesaa--resooingg soogg-pro-hiie -iigh (reasoning). reported in Figure E.3. We find that though both systems are based on the same search module, the reasoning model cites more community and blog sources and less Wikipedia articles, compared the non-reasoning model. These results are consistent with our correlational analysis of user preferences and cited domains (e.g., users rejecting Wikipedia sources in specific settings).
We also control for all features simultaneously (response length, number of citations, and citation sources) to compute adjusted model scores. The scores and rankings before and after applying these controls shown in Figure E.4. We observe that the model scores and rankings tend converge after the controls are applied. This convergence is particularly evident within model families: the top three models from Perplexity show significant convergence. This suggests that much of the variation between models in the same family is accounted for by the control features. We note that features such as factuality and coverage may be correlated with the number of citations or specific citation this Ahissbaa AN
toyrcegrye w
As described in Subsection 3.2 and illustrated in Figure 7, we analyzed citation attribution on approximately 100 conversations per intent category. This section outlines the full pipeline.
We first sampled around 100 conversations from each intent category, excluding the "Other" class. To retrieve the cited content, we used Firecrawl' as our default batch scraping tool. For social media platforms other domains where Firecrawl could not be applied, such as Reddit and YouTube, we leveraged official APIs or extracted video transcripts when available. For the remaining URLs that failed due technical limitations, access restrictions, licensing concerns, we excluded the corresponding samples. Overall, we took care to ensure ethical data usage, relying only on publicly accessible content and following best practices regarding rate limits and terms of service. In total, we collected over 20,000 web documents across 780 conversations.
Next, we used gemi n 22. - www.h--reeiee--eeeeeepe 0417 7 to analyze the data, selected for its strong performance on long-context reasoning. reliable structured (JSON) outputs, and reasonable cost. For each conversation, we first used the model to parse user messages into (claim, URL) pairs, following the structure shown in Figure 7(a). Then, for each pair, we provided the scraped markdown content and the associated claim to infer attribution judgments. Each attribution was labeled as Support. Irrelevant, or Contradict. The prompt used for LLM--bsee tagging shown on the next page.
https//www.frrcrrw..ddyy
34

---

Published as a conference paper at ICLR 2026
gemini-2.5-pren S-pro-groumding
gemini-2.5-fiessh 5-iistt----gg
gemini-2.0 2.0-fi-h rissh-grounding
sonar-reasoninn-pro
1125
soner-pro-high
soner-reasoning
1100
got search high
Arena Score
1050
1025
1000
original
controlled
Figure E.4: Model Scores Before and After Control. Model scores and rankings converge after the controls are applied.
As illustrated in Figure 7(b), we aggregated the number of supporting, irrelevant, and contradicting claims per turn and used these counts as features in the Bradley-Terry model, following the same setup as the other control experiments. Human experts validated subset of the outputs. Future work further scale this pipeline add onto analysis methodology.
LLM Prompt for Claim-Citation Attribution Assessment
Task: Tell if the clain be verified supported by the web content.
Specifically:
If the claim is supported by the web content, return "support" If the claim contradict to the web content, return "contradiat" If the claim is completely irrelevant to the web content, return "irrelevant"
Claim: (claim)
Web content: (web content;
Return in the jsen format:
"ressoning"; "cexplain reasonings",
"answer"; <"support" "contradict" "irrelevant">
F CROSS-SETTING ANALYSIS
In this section, extend our analysis of Subsection 3.3 and study model performance changes across different benchmarks. Specifically, we explore the following research question: How does model performance and ranking change across different search and non-search benchmarks?
For offline evaluation of models in search settings, use BrowseComp (Wei et al., 2025) and SimpleQA (Wei et al., 2024)². For testing the general performance the models in non-search settings, ArenaHard-v2 (Li et al., 2024). BrowseComp and SimpleQA assess factuality based
Due the high cost of running search-augmented I.IMss, evaluated each model the same randomly sampled subset of 500 questions. The subset sampled once and shared all models.
35

---

Published as a conference paper at ICLR 2026
on well-specified ground truths, while ArenaHard-v2 utilizes an LLM-as-a-judge framework set of challenging prompts filtered from Chatbot Arena's Text Arena dataset. Although these benchmarks test different aspects of standard and sarrchauugmntee LLMs, all are partially captured by the Search Arena prompt distribution shown in Figure! 1.
The selected models have near-zero accuracy on the BrowseComp benchmark; this finding is not surprising, as the questions are specifically designed to evaluate and challenge deep research pipelines. Arena, SimpleQA, and ArenaHard-v2 ar-are
The model scores for the rest of the benchmarks-Sean provided in Table F.1. Additionally, we calculate model win rates two subsets of the Search Arena dataset based on the annotated intent classes (see Section 2)-Search Arena SFact+ includes only Factual Lookup and Info Synthesis queries, while Search Arena (Other) contains the remaining subset. We use Kendall's tau for comparing ranks and Pearson correlation for comparing raw scores across benchmarks. On SimpleQA, model accuracy is saturated (ranging from 89% to 93%), with minimal separability between the models and low agreement with Search Arena (r = 0.422, = 0.582). In contrast, ArenaHard-v2 shows greater performance variance and separability, and has higher agreement (r 0.556, 0.844) with Search Arena compared to SimpleQA. However, the rankings differ; notably, the three reasoning models rank lower in ArenaHard-v2, suggesting that, while combining web search with reasoning improves performance in Search Arena, it may degrade performance on ArenaHard-v2 prompts. Additionally, when comparing the two subsets of the Search Arena with ArenaHard, Search Arena (Other) has higher agreement (r 0.644, r= 0.882) compared to Search Arena (Fact+Synth) (7 = 0.511, r = 0.675). This finding is expected as the prompt distribution in Search Arena (Other) is closer to that of ArenaHard (e.g., creative writing, problem-solving).
Furthermore, we compare model performance with and without search on the SimpleQA and ArenaHard-v2 v2 benchmarks. We used Gemini models in this case study, search is implemented as a tool (GoogleSearch tool) and can be casily turned on and off. The performance change is shown in Figure F.2. Search significantly improves model performance SimpleQA (models' performance saturates at around 90%). However, performance degrades on ArenaHard-v2, with non-search variants achieving higher scores.
Table F.1: Model scores across three benchmarks Search Arena (win rate), SimpleQA (accuracy), and ArenaHard-v2 (win rate). On SimpleQA, models' performance is saturated and lead to minimal separability in model scores. Search Arena and ArenaHard v2 provide more separability, but the rankings are different.
Model
Search Arena
Search Arena (Fact.)Synth.)
Seurch Arena (Other)
SimpleQA
ArvaHHard-12
sonar ressoning seciiing-rr--iaee pro-high
66.6 (22.1/222)
(33.3/+333))
65.0(2..1+2.7))
91.5
241.7
gemini-2.5 pro-gnt gr-ggrrudddng
66.7 (+2.5/+2.4)
(-3.8
65.3 (:3.2/ +3.2)
90.6 (2.4)
33.5 (1244
somat-pr--high
63.7 (-2.2) +2.4)
(-3, (-3.5/434333) /43.3)
(62. +2.)
93.2
(-2.1-21/222..
sonar reasoning
(-2.5/+355))
(-3.4
61.3 (3..++33)
(26(22.4+2.2)
29.1
(-24|2.3))
sonar pro sonar
57.5 +2.8)
56.8 (4.2)
58.4 (-3.9 +4.0)
92.(2241
39.9 (22.2
55.8 / (2.8)
54.1 (43.8)
56.8-399/ +3.8)
918
(-..9-11972))
gemini-2.5 2.0-iii-2--0ass---ooooo ---oasi-grounding
/+3.5)
(4.5 +5.8)
46.1 (-18//447
(-.66226)266))
2-1.4-1/+11
gemini-2.. Bash grounding
41.7
+2.9)
42.8 +4.2)
41.0 +3.4)
13.4 (-1.0/+1..)
got- ser--o--erch-pee preview-high
(-2.5-2.5 42.5)
36.2 (-3.6/ (3.8)
(-3.5 +3.2)
(-2((28/42.6)
16.6 (-1./4412))
gol-4o-mini- co-miin--eee prevce
29.3
(0.7
(-.66
(2.6
G ADDITIONAL COMPARATIVE ANALYSES
G.1 CORAL
As our setup is similar to prior work in conversational search, we conduct additional comparative analysis with CORAL (Cheng al., 2025a), a benchmark designed to assess RAG systems in multi-turn conversational settings. Compared to CORAL, Search Arena conversations are collected from natural open-ended interactions between humans and search-augmented LLMs and contain user-provided preference labels.
Similar to the datasets studied in the main paper, CORAL focuses on English-only conversations, whereas the Search Arena prompts cover diverse set of 71 languages. Search Arena (24k) also is larger in scale compared to CORAL (8k). Furthermore, in contrast to SimpleQA (Wei et al., 2024) and BrowseComp ( (Wei et al., 2025) and similar Search Arena, CORAL includes multi-turn conversations. Compared to Search Arena (Figure B.3), CORAL conversations are longer on average (Figure G. Left). In terms of prompts, CORAL focuses on information-seeking queries only,
36

---

Published as a conference paper at ICLR 2026
reasoninnnng pro-tigh
Search
gemmi ppro-gruuntinggg
sonampre pro--hhhh
konar
gemmi flash grounding
get wew.seerth-pprieeeeeee
4p-m4ni prevee
Search Arena (Other)
Search Arena (Fact+Synth)
100
Arenallard
Smmpl@g
Figure F.1: Model Scores Across Benchmark, (1) Models' performance on SimpleQA is saturated with minimal separability. (2) Performance variance in ArenaHard-v2 is comparable to that of Search Arena, but the rankings are different, with reasoning models having higher performance in search settings. (3) Model scores and ordering in Search Arena (Other) closer that of ArenaHard compared to Search Arena (Fact+Synth).
55
gmmpt-2.5ppr
50
search non-search
45
pemini225-foa
Fitt y -0.20k 43,74
ArenaHard Score
40
35
3-350
-0
25
gemmi2255 flssh
20
30
70
SimpleAA Score
Figure F.2: SimpleQA vs ArenaHard-v2 Performance. Search and non-search performance of Gemini models SimpleQA and ArenaHard-v2 benchmarks. Search improves performance SimpleQA, while degrades the score on ArenaHard-v2.
whereas Search Arena covers broader range of intents (c.g.. factual lookup. creative generation). The prompt length distribution Search Arena prompts also more varied compared to CORAL, with the latter containing more concise questions (see Figure G.1, Right).
Additionally, in contrast to Search Arena, CORAL explicitly constructs ground truth answers, assum- ing bounded set of documents (including golden). Search Arena focuses human preferences, which, as observed in Section 3, do not need to correlate with factuality and quality of the retrieval. Finally, Search Arena data is collected from real-worle natural interactions, enabling us to study real-world usage of the models and analyze how model response features correlate with the users' pref- erences. Thus, the differences between the two datasets are due to the differences in the corresponding intents, use cases, and motivations.
G.2 WILDCHAT
Similar to the Text Arena and Search Arena datasets studied in the main paper, WildChat (Zhao et al., 2024) is crowd-sourced human-LLM conversation dataset. Similar the Text Arena and in contrast to the Search Arena, the interactions in WildChat are with regular LLMs without access to additional tools, such web search.
37

---

Published as a conference paper at ICLR 2026
Despite the differences in the settings, we compare WildChat with Search Arena across multiple axes; we use a random sample (100k) of the non-toxic filtered version (3.2M) of the original dataset for the comparative analysis below. First, the full WildChat dataset is much larger in scale (4.8M), compared to Search Arena (24k). Both datasets are multilingual, containing interactions with over 70 languages. Turn and prompt length distributions of the WildChat dataset are provided in Figure G.2. WildChat turn count distribution is similar to that of Search Arena, with Search Arena containing slightly more multi-turn conversations. Additionally, WildChat prompts are longer on average (594.9) compared to those of Search Arena (57.1). This difference arises from the difference in the user intents across the two datasets: as reported in the Text Arena comparative analysis, search- enabled settings significantly impact the intent of the queries (e.g., users more likely to ask real-time infornation-seeking questions).
Search Arena
CORAL
Search Arena mean: 57.1
16-2
CORAL mean:
Density
10-
10aa
104
200
400
500
800
1000
1200
1400
Turms per Donsemation
Prompt Length (words)
Figure G.1: (Left) CORAL conversation length (number of turns) distribution; CORAL conversations are longer than those in Search Arena, with 8.3 turns on average. (Right) CORAL prompt length distribution; CORAL prompts (red) are more concise and less diverse (lower variance) compared to Search Arena prompts (blue).
10
Search Arena
WildChel
$346
Density
296
2:16
100
400
Turns per Conseratiion
Prompt Length (words)
Figure G.2: (Left) WildChat conversation length (number of turns) distribution; WildChat conversa- tions similar in distribution to Search Arena Figure B.3, with slightly fewer multi-turn samples. (Right) WildChat prompt length distribution; WildChat prompts (pink) are longer compared to those of Search Arena (blue).
38


Reviews:
[{"review_id": "nh48oUavsn", "reviewer": "Reviewer_7PyT", "summary": "This study aims to solve the issue of evaluation benchmark for search-augmented LLM limited in scale and narrow in scope, often constrained to static, single-turn, fact-checking questions. To this end, a crowd-sourced, large-scale, human-preference dataset is proposed together with the analysis and performance accessing across different categories and settings. Overall, this is an important contribution in evaluating retrieval-augmented generation.", "questions": "1. What is the potential mechanism to improve the reliability of the constructed datasets based on human votes?\n\n2. What is the difference between search-augmented LLM and retrieval-augmented generation (in conversational scenarios)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Unlike prior datasets such as SimpleQA and BrowseComp which are static, English-only, single-turn fact-seeking queries, the proposed Search Arena evaluates models in diverse, open-ended, multilingual, and multi-turn settings.\n\n2. The human-preference analysis are detailed, covering number of citations, supportive claims, cited sources, etc. This is the crucial point in terms of the construction principle of Search Arena.\n\n3. A set of detailed experimental results and analysis are provided to cover various aspects.", "weaknesses": "1. The reliability of the collected data should be further judged, although the more sophisticated approaches are expensive.\n\n2. The definition of search-augmented LLM and retrieval-augmented generation should be further distinguished if the author discusses them in different context as shown in the related work.  Besides, the conversational search [1,2] is highly related to the proposed Search Arena, i.e., multi-turn human-AI interaction in search setting.\n\n3. Section 2 discuss the difference compared to SimpleQA, BrowseComp, and Text Arena, but lack of the comparison with existing conversational search/RAG datasets, e.g., TopiOCQA [3], Coral [4]. \n\n[1] Neural Approaches to Conversational Information Retrieval. SIGIR 2020.\n\n[2] A Survey of Conversational Search. ACM TOIS 2025.\n\n[3] TopiOCQA: Open-domain Conversational Question Answering with Topic Switching. TACL 2021.\n\n[4] CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation. NAACL 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iruBr2PHD4", "reviewer": "Reviewer_ttYs", "summary": "The study introduces the first large-scale, crowd-sourced dataset of human interactions with search-augmented LLMs. It analyzes how users engage with search-augmented LLMs and finds that user preferences are strongly influenced by citation quantity and source type (even when citations are irrelevant). A cross-arena experiment shows that web search improves model performance in both search and non-search settings, but models without search capabilities perform poorly when users expect real-time retrieval.", "questions": "The reasoning traces shown in Figure 3 are quite interesting (demonstrating multi-document analysis, filtering, and synthesis). Did the authors observe cases where reasoning models misinterpreted their sources? If so, how might such errors in reasoning be detected at a larger scale?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "-The study is impressively comprehensive, drawing on a diverse dataset that spans 11,650 users across 136 countries, 13 models, and 70 languages.\n\n-The authors conducted a series of insightful and relevant analyses on human preferences in search-augmented LLMs, such as the “Types of Cited Sources” section. The finding that Wikipedia citations correlate negatively with user preference (while social media and community sources correlate positively) is both counterintuitive and well-explained (i.e., Wikipedia’s breadth and outdatedness often reduce perceived relevance).\n\n-Impactful work: This work releases a large-scale, human-preference dataset for search-augmented LLMs, which acts as a stepping stone for future research works to understand how users evaluate search-augmented LLMs.\n\n-The paper is well-structure and well-motivated, with the intent diversity section drawing clear comparisons to prior datasets. The writing is polished, and the narrative is easy to follow.", "weaknesses": "The current premise of search arena relies on human preference signals, but the paper’s own findings cast doubt on whether these signals are reliable indicators of true search quality. In particular, the Citation Attribution analysis shows that users often fail to distinguish between supporting and irrelevant citations and tend to prefer responses with a higher number of citations (regardless of the validity)). Human preference may conflate perceived credibility with actual factual correctness, leading to rankings that reward citation quantity or presentation style rather than true retrieval or reasoning quality.\n\nThe authors could expand their Citation Attribution experiment (currently only limited to ~100 examples) as a more objective evaluation metric. In particular, metrics assessing (i) citation accuracy, (ii) citation relevance (semantic alignment between retrieved source and response content), (iii) recency (temporal freshness of cited information), and (iv) retrieval rarity or specificity (the model’s ability to surface rare but authoritative sources) could provide a more faithful measure for search-augmented LLM performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rycrk0WLDW", "reviewer": "Reviewer_Uoa4", "summary": "The paper introduces Search Arena, a crowd-sourced dataset for analyzing search-augmented language models collected by deploying a chat platform to collect in-the-wild user interactions. The dataset contains >24k paired multi-turn conversations and over 12k human preference votes, collected from interactions with 13 different models. This work addresses limitations of existing factoid benchmarks that are focused on single-turn, fact-checking questions. The dataset spans 71 languages and presents a new user intent taxonomy. \n\nThe authors use this dataset to analyze user preferences and find that users prefer longer responses and more citations. An interesting finding is that this preference for more citations holds even when the cited content is irrelevant to the claim. The paper also presents a cross-arena analysis, where they show that search-augmented models perform comparably in general-purpose chat settings, but models relying only on parametric knowledge underperform in search-intensive settings.", "questions": "- The authors mentioned that reasoning models filter irrelevant content and cite fewer sources. Did you find any difference in the types of sources cited by reasoning vs non-reasoning models?\n- It is unintuitive to me that citing Wikipedia is negatively correlated with user preference. Since it is hard to control the prompt set while just varying the cited sources, I wonder if the coefficient differences between cited sources are actually meaningful to draw conclusions from?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-motivated and easy to follow. It addresses a clear gap, i.e. the lack of large-scale multi-turn and multilingual datasets for evaluating search-augmented LLMs.\n- The Search Arena dataset can be very useful to researchers working on a broad set of topics. Its scale as well as diversity in languages and query intents make it a valuable resource.\n- The analysis of user preferences is interesting and results in some non-obvious findings. The finding that user preference is positively correlated with the number of citations, even when those citations are irrelevant / do not support the claims, is concerning and could affect model design decisions.\n- The cross-arena analysis is interesting as well because it provides strong evidence that search augmentation is beneficial in search-heavy tasks without degrading general chat performance.", "weaknesses": "- The paper presents a lot of correlational evidence, for eg, about which response features and cited sources users find important, but it states these findings more definitively than the correlational analysis supports. I think this is a major weakness and I would suggest rewriting these claims with caution.\n\n- The paper relies heavily on LLM-based pipelines for analysis, particularly for user intent classification and citation attribution. For the citation attribution validation, the process is described vaguely. For eg, the paper states that \"Human experts validated a subset of the outputs\" but does not provide inter-annotator agreement scores or a quantitative comparison between the LLM pipeline and human judgments. Given that the finding about irrelevant citations is a significant takeaway, a more robust validation of this pipeline is necessary. \n\n- Relatedly, the citation attribution pipeline was run on just 100 English conversations per intent category due to cost / scraping challenges, which is a pretty small subset. It is unclear if these findings about irrelevant citations generalize across all languages and the full dataset.\n\n- The paper provides limited discussion of some relevant prior work, such as WildChat and WildBench.\n\n- The paper offers limited to no discussion on precisely how future work should utilize the Search Arena dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KpPcXDd5gI", "reviewer": "Reviewer_Tf64", "summary": "This paper introduces Search Arena, a large-scale, multilingual and multi-turn benchmark for search-augmented LLMs that contains 24k real user conversations and 12k human-preference judgements. The authors analyse how response length, number of citations, source types and citation fidelity correlate with user preference, and conduct cross-arena experiments to quantify the gain of web access. The dataset and ranking pipeline are released under CC-BY.", "questions": "- What concrete steps will you take to handle residual PII or sensitive queries that automatic filters missed, and will you offer users a mechanism to retroactively remove their data?\n- For groups that wish to evaluate a new model on Search Arena without running fresh human battles, which fully automatic metric pipeline do you recommend, and how should statistical significance be computed when only LLM judges are available?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The benchmark itself is a valuable contribution: it is two orders of magnitude larger than prior search-LLM datasets, covers 70+ languages, multi-turn context and nine diverse intents, and provides full system traces (query, retrieved pages, response, human vote).\n- The quantitative analysis offers actionable insights—e.g. users reward more citations even when irrelevant, Wikipedia citations are penalised, and reasoning models filter sources more aggressively—all statistically validated with a Bradley-Terry model.", "weaknesses": "- As an evaluation benchmark the dataset is still tied to human preference. Automatic metrics are only sketched (e.g., claim-citation labels), so new models must either run costly human battles or risk LLM-as-a-judge drift. The paper also does not demonstrate any model-training experiments that exploit the released supervision.\n- The main qualitative insights—citation count boosts perceived credibility, users distrust Wikipedia, etc.—largely echo existing insights; the work does not surface novel, counter-intuitive findings.\n- Although ethics are discussed, the entire corpus is mined from users who clicked “accept” once. No right-to-withdraw, no IRB statement, and limited sensitive-content filtering leave lingering privacy concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)