Towards Personalized Deep Research: Benchmarks and Evaluations
Authors: Yuan Liang
Jiaxian Li
Yuqing Wang
WANG PIAOHONG
Motong Tian
Pai Liu
Shuofei Qiao
Runnan Fang
He Zhu
Ge Zhang
Minghao Liu
Yuchen Eleanor Jiang
Ningyu Zhang
Wangchunshu Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: Personalization
benchmark
Deep Research
Agent
Primary area: datasets and benchmarks
TL;DR: Introducing PDR-Bench and the PQR framework, the first comprehensive benchmark for evaluating personalization in Deep Research Agents.

Abstract: Deep Research Agents (DRAs) can autonomously conduct complex investigations and generate comprehensive reports, demonstrating strong real-world potential. However, existing evaluations mostly rely on close-ended benchmarks, while open-ended deep research benchmarks remain scarce and typically neglect personalized scenarios. To bridge this gap, we introduce Personalized Deep Research Bench (PDR-Bench), the first benchmark for evaluating personalization in DRAs. It pairs 50 diverse research tasks across 10 domains with 25 authentic user profiles that combine structured persona attributes with dynamic real-world contexts, yielding 250 realistic user-task queries. To assess system performance, we propose the PQR Evaluation Framework, which jointly measures Personalization Alignment, Content Quality, and Factual Reliability. Our experiments on a range of systems highlight current capabilities and limitations in handling personalized deep research. This work establishes a rigorous foundation for developing and evaluating the next generation of truly personalized AI research assistants.

Published as a conference paper at ICLR 2026
TOWARDS PERSONALIZED DEEP RESEARCH: BENCHMARKS AND EVALUATIONS
Yuan Liang Jiaxian Li Yuging Wang Piaohong 2 .Moongg Tian Pai Liu Shuofei Qiao*, Runnan Fang* He Zhu Ge Zhang* Minghao Liu
Yuchen Eleanor Jiang Ningyu Zhang*; Wangchunshu Zhou of
OPPO Zhejiang University MM-A-P 2077.A1
(liang zhangningyul jzju.. edu cn, zhouwangghhunsssuuuppoooom
ABSTRACT
Deep Research Agents (DRAs) can autonomously conduct complex investiga- tions and generate comprehensive reports, demonstrating strong real-world po- tential. However, existing evaluations mostly rely on close-ended benchmarks, while open-ended deep research benchmarks remain scarce and typically neglect personalized scenarios. To bridge this gap, we introduce Personalized Deep Re- search Bench (PDR-Bench), the first benchmark for evaluating personalization in DRAs. It pairs 50 diverse research tasks across 10 domains with 25 authentic user profiles that combine structured persona attributes with dynamic real-world con- texts, yielding 250 realistic user-task queries. To assess system performance, we propose the PQR Evaluation Framework, which jointly measures Personalization Alignment, Content Quality, and Factual Reliability. Our experiments on a range of systems highlight current capabilities and limitations in handling personalized deep research. This work establishes rigorous foundation for developing and evaluating the next generation of truly personalized AI research assistants
INTRODUCTION
Recent advances in large language models (LLMs) have enabled the development of Al agents capa- ble of conducting complex deep research. Early LLMs focus isolated tasks like QA and transla- tion, later advancing with tool integration for autonomous information retrieval and synthesis. More recently,
class of advanced systems has emerged, known Deep Research Agents (DRAs), including industry solutions (OpenAI, 2025b; Google DeepMind, 2025; xAI Team, 2025; Perplexity Team, 2025; Moonshot AI, 2025; Batee 2025b) and open-source systems (Li et al., 2025b;e; Zhu et al., 2025a; Zhou et al., 2023; 2024; Wang et al., 2025a: Hu et al., 2025; Manus AI, 2025; MiroMind AI Team, 2025; ByteDance, 2025a; et al., 2025a; Tang al., 2025; Shi et al., 2025; Zhu et al., 2025b). DRAs extend LLMs by incorporating dynamic reasoning. adaptive planning, and iterative tool use to acquire, aggregate, and analyze external information (Huang et al., 2025), thereby enabling end-to-end research workflows and the production of structured, comprehensive reports.
Despite these advances, to fully realize the potential of these intelligent systems in everyday human contexts, they must able to adapt their behaviors and interactions to the specific needs of dif- ferent users (Fischer, 2001; Kirk et al., 2024; Rafician & Yoganarasimban, 2023), quality known as personalization. Important real-world decisions, from choosing a vehicle to making an invest- ment, strongly influenced by user's unique needs, preferences, budget, and prior knowledge. In these scenarios, the agent's value lies not only in generating a comprehensive report, but also in acting personalized assistant that tailors its information filtering, reasoning, and recommenda- tions. However, this critical dimension of personalization is a major blind spot for current evaluation methodologies.
Equal Contribution.
1 Corresponding Author. Code; https://gitthie hPwpm//PPPPPRP Personalssa..... A ize wewpeeeeeeeeeeeeeeeeeeeeeeeeeeee

---

Published as a conference paper at ICLR 2026
Existing deep research benchmarks, including close-ended suites like GAIA, BrowseComp. HLE, and X- X-Bench (Mialon et al., 2023; Wei et al., 2025; Phan et al., 2025; Chen et al., 2025a) and open- ended ones like DeepResearch Bench, ResearcherBench, and DeepResearchGym (Du et al., 2025; Xu et al., 2025; Coelho et al., 2025); focus exclusively on factual accuracy and comprehensiveness, failing to assess user-specific adaptation. Conversely, existing personalization benchmarks such as LaMP, PersonaGym, Personalens and PersonaFeedbbck (Salemi et al., 2024; Samuel et al., 2025; Zhao et al., 2025; Tao et al., 2025) are confined narrow domains like dialogue or recommendation and do not address the complex deep research. To the best of our knowledge, our work is the first to systematically incorporate personalization into the evaluation of DRAS, filling critical gap in current research.
To address this gap, we introduce Personalized Deep Research Bench, novel benchmark specif- ically designed to evaluate personalization in deep research agents. Our benchmark provides a rigorous framework for assessing how well agents can integrate user profiles into their research workflows, and whether their outputs are not only comprehensive and accurate, but also tailored and practically useful for the end user. By formalizing and evaluating this missing dimension, our work paves the way for the development of more effective and genuinely personal AI assistants.
Our main contributions are summarized as follows:
We formally introduce the task of personalized deep research, which extends beyond generic information synthesis by requiring DRAs to adapt retrieval, reasoning and reporting to user personas.
We propose Personalized Deep Research Bench, the first benchmark specifically targeting personalization in DRAs It consists of 50 diverse tasks that span 10 domains and are paired with 25 real-world user profiles, yielding 250 unique user-task pairs, enabling systematic evaluation of both task complexity and persona-driven adaptation.
We develop the PQR Evaluation Framework, novel and comprehensive methodology that evaluates generated reports along three orthogonal dimensions: (P) Personalization Alignment, (Q) Content Quality, and (R) Factual Reliability, providing holistic measure of agent utility in real- wolld research scenarios.
We conduct extensive experiments across a broad spectrum of open-source DRAs, com- mercial deep research systems, LLMS with search tools and advancing memory systems, revealing both strengths and limitations in handling personalization.
2 RELATED WORK
2.1 EVALUATING DEEP RESEARCH CAPABILITIES
Evaluating DRAs requires benchmarks that go beyond traditional QA tasks to assess multi-turn retrieval, tool use, and structured report generation. Close-ended benchmarks such as GAIA, BrowseComp, HLE, and X-Bench (Mialon al., 2023; Wei al., 2025; Chen al., 2025a) of- fer controlled evaluations, yet rely on synthetic tasks and fall short of reflecting the challenges of authentic research scenarios. Recently, open-ended deep research benchmarks have been proposed to specifically evaluate deep research capabilities. DeepResearch Bench (Du et al., 2025) offers 100 PhD-level tasks across 22 fields, introducing the RACE and FACT frameworks for report quality and retrieval assessment. Mind2Web 2 (Gou et al., 2025) features 130 real-world tasks with live web browsing and proposes the Agent-as-a-Judge framework for automated correctness and attri- bution. ResearcherBench (Xu et al., 2025) focuses on 65 frontier Al questions across 35 subjects with a dual rubric-factual evaluation. Additionally, BrowseComp- (Chen et al., 2025b) extends BrowseComp (Wei et al., 2025) by pairing each query with curated documents and challenging neg- atives to isolate retriever and LLM contributions. DeepResearchhymm (Coelho et al., 2025) provides an open-source sandbox with reproducible search APIs and standardized protocols for transparent, low-cost benchmarking. Nevertheless, these benchmarks focus on general research capabilities and lack metrics for personalization the alignment of research with user-specific goals and preferences.
2

---

Published as a conference paper at ICLR 2026
Deep Research Task Formulation
Authentic User Profile Construction
Personalized Query Formultion
Domaina
DO
Education
Carner
Deep
Teak Selection
Annotators
Spcoification
Structured Schame
oge, gender, education Scoupstion, Family, peof freence
Explon Per Persona
User Profile
Committe Review
Task Design
User 1
Parsorulized
tge 20, gander male, educction ptudent mejor in
Personas
Personalized Deep Research
Ib
Topkl: 3,
Domai: Edunstion
beckground Tiplon
Task: wont advance educshoral
User User inferoctions phone
Tork2t User 2,
Tesk
Teck
Tesk
Personaliedd
Pairs
Figure I: Benchmark Construction Pipeline: (1) Design 50 deep research tasks across 10 domains; (2) Build authentic user profiles from 25 volunteers; (3) Generate 250 personalized task-user pairs.
2.2 BENCHMARKING PERSONALIZATION PERFORMANCE
Meanwhile, most personalization benchmarks focus on general tasks and remain insufficient for complex deep research scenarios. LaMP (Salemi et al., 2024) introduces seven classification and generation tasks evaluate the personalized output capacity LLMs. PersonaGym (Samuel al., 2025) introduces PersonaScore to evaluate the adherence of LLM agents to assigned personas at scale. Personall.LM (Zollo et al., 2025) uses reward models act as different user personas to evaluate response preference. AI Persona (Wang et al., 2024b) concentrates on the lifelong learn- ing of user profiles with LLM-as-a-judge evaluation. Additionally, PersonaMem (Jiang et al., 2025) benchmarks the adaptability of LLMs to evolving user personas. PersonaFeeedback (Tao et al., 2025) provides a large -uman-annotated benchmark for response tailoring to explicit personas. Person- aLens (Zhao al., 2025) introduces LLM-based user and judge agents to assess personalization and task success in realistic dialogues.
Overall, current benchmarks either neglect personalization fail to capture the complex nature of deep research, highlighting the pressing need for a new benchmark specifically designed to measure the personalized performance of DRAs.
3 BENCHMARK CONSTRUCTION
To rigorously evaluate the personalized capabilities of deep research agents, we introduce the Per- sonalized Deep Research Bench, benchmark designed to mirror the real-world personalized deep research scenarios. Its construction is grounded in two key components: diverse set of deep re- search tasks and a collection of authentic, multifaceted user profiles, as shown in Figure
3.1 DEEP RESEARCH TASK FORMULATION
Domain Specification and Task Generation. To begin, we defined set of 10 distinct domains, D {dz, d2, d covering major and impactful aspects of daily life (e.g., Career Development, Education, Healthcare, Financial Planning). To ensure that the tasks within each domain are both realistic and practically relevant, we collaborated with a diverse group of domain experts, such as travel bloggers, financial advisors and educational consultants, to design the initial set of tasks.
Committee Review and Validation. Each task underwent multistage validation by commit- tee of Master's/Pai s/DD researchers, data scientists, and product managers, following three principles: Complexity (t): requiring multi-step reasoning, retrieval, and analysis; Clarity (f): unambiguous
3

---

Published as a conference paper at ICLR 2026
descriptions with clear objectives; Alignment (t): supporting the scenarios of personalized deep research.
Finally, we systematically formulated 5 balanced tasks per domain, yielding 50 tasks in total: T t i 1,...,50 50, with t, = (qi d(t+)) where qi is the query and d(t,)) 'D the domain. A parallel English set TEN was also created, semantically aligned with the Chinese tasks.
3.2 AUTHENTIC USER PROFILE CONSTRUCTION.
A key innovation of our benchmark lies in the careful design of highly realistic and richly detailed authentic user profiles. We moved beyond synthetic stereotyped characterizations by grounding our profiles in real user data.
Structured Explicit Persona Collection. We recruited 25 volunteers with diverse demographic profiles across age, profession, income, and life stage. After receiving standardized training on data authenticity and privacy, volunteers mapped their authentic personal details onto specially designed persona schema, S, which can be found in the Appendix D. This process yielded set of 25 structured explicit ground-truth personas Ps, denoted as: P. (Pxj j == =1,.,... 2,....,5]..
Dynamic Personalized Context Integration. To complement these explicit personas with dy- namic context, we employed professional annotators simulate the daily interactions of these col- lected personas through phone APP. Over a period, they were instructed to: 1) Record naturalistic memory snippets (m,), such travel aspirations, health goals, and family plans; and 2) Conduct conversational interactions (c₂) with the intelligent assistant integrated in the app. This longitudinal data captures each user's evolving interests, habits, and implicit preferences. These multi-modal data streams were then processed by the built-in management system of the APP, fa. to generate dynamic personalized contexts: P. {Pej Pe₁ fe(m,, c2), j I,..., 25). Annotation details are in Appendix I.
For convenience, we define the complete user profile set as the collection of paired structured explicit personas and dynamic personalized contexts:
P {(Ps). Paş) L,...,223 25}
3.3 PERSONALIZED DEEP RESEARCH QUERY FORMULATION
The final stage of benehmark construction involved the principled pairing of user profiles with deep research tasks to generate meaningful, personalized queries. We recognized that random pairing would fail to capture the intrinsic relevance between a user and their research needs.
To address this, employed a user-driven, committee-guided alignment protocol. Each of the 25 volunteers first reviewed the full task pool T and selected tasks that were personally relevant. Then, the committee curated and refined these selections through rigorous discussions, ensuring: (1) diversity of user profiles associated with each task, and (2) overall alignment between each user-task pair. This process yielded a user subset P; C P for each task 4, where [P, 5.
Finally, a total of 250 personalized personalized deep research queries were formed:
Q (p, 1....,,.,. 50, E P.). |Q| = 250
where each query combines one high-quality deep research task t with corresponding profile p from its assigned user set.
This benchmark faithfully mirrors real-world personalized deep research scenarios while providing a standardized, reproducible, and scalable evaluation setting. By jointly modeling task complex- ity, authentic user profile diversity, and motivational alignment, it provides rigorous testbed for evaluating whether agents can effectively integrate user profiles into deep research and deliver truly personalized high-quality outputs.
4 EVALUATION METHODOLOGY
How do end-users judge the value of a Deep Research report? "Is this report for me?" (Personal- ization), "Is it w (Quality), and "Is the information true?" (Reliability). Existing eval-
4

---

Published as a conference paper at ICLR 2026
A...
Deep Research
Personalization Alignment
LLM
Task
User
B.b Ortaroon Gemeration
Judge LLM
Generated Report
Whetlar ached lunh hetioe/ clisdy march the uee's prefile
Gool Alignment- 0.16
natch: firting clllnced
Deep Reseerch Agents
Content Alignment: 0.39
Starel: 3,5/10
Presentation Fit: 0.33
key cdmiasion festers workingg CWee
Andyysii anc Ssored S/1p
Actionability Actiendbilty&Paaciiolty 0.12
Generated Report
Omerrtoon Weggh Alocaatin
Quality
Factual Reliabillty
Judge
Guiii support
Teak
Generated Report
Generated Report
LLM
Extrazion
GuinZ unsupport
Depth & Irdight: 0.45
G-Score
Logical Ccherency:
Claimi: Monthly living Boston about $1,800 source (di [1], sourcel /ww.p////w......... &
Bub Crizzion Generaton
Clarity Readiability: 0.32
soct
X006 scurce Xoc pources xxo
Whet Search
Content
Score
Figure 2: Overview of the PQR Evaluation Framework A Multi-Dan Doatii-Dimenionall Assessment Sys- tem, Integrating Personalization Alignment (P), Content Quality (Q), and Factual Reliability (R).
uations, however, typically stress report quality or factual correctness, neglecting personalization. To systematically address these core concerns, we propose the PQR Evaluation Framework, novel and comprehensive methodology assessing reports along three complementary axes: (P) Per- sonalization Alignment, (Q) Content Quality, and (R) Factual Reliability. This joint consideration provides a holistic, user-centered assessment of Personalized Deep Research, as shown in Figure 2.
4.1 P PERSONALIZATION ALIGNMENT
Evaluating the personalization of generated report is significant challenge due to its subjective, multi-dimensional nature. Recent studies (Wang et al., 2024a; Guan al., 2025; Zhu al., 2025c) consistently emphasize that personalization evaluation should move beyond global correctness to- ward preference-aware and user-centered assessment paradigms, highlighting the need for individ- ualized criteria rather than generic evaluation frameworks. Motivated by these insights and our re- quirement analysis, we introduce Personalization Alignment (P-Score), a dynamic evaluation frame- work that generates customized criteria and scores for each user-task pair.
The framework is built on four fundamental dimensions: Goal Alignment (GOAL), Content Align- ment (CONT), Presentation Fit (PRES) and Actionability & Practicality (ACTI). Detailed def- initions of dimensions are provided the Appendix E. The P-Score is computed via a three-stage, LL-M-driven pipeline that operationalizes these dimensions into quantitative score:
Stage 1: Dynamic Dimension Weight Allocation. An LLM, acting meta-evaluator, analyzes the input task T and user persona P. determine the relative importance of the four dimensions. This stage outputs weight vector W (wa de for the set of personalization dimensions Dp [Goal Alignment, Content Alignment, ...). where Eder, Wa = 1.0.
Stage 2: Granular Sub-Criterion Generation. For each dimension d Dp, the LLM generates a set of granular sub-criteria Cf2² (Ci..... }, again conditioned on the task T and user persona P.. Each sub-criterion e; is assigned a weight We, such that D We. 1.0.
Stage 3: LLM--Poered Scoring. A separate LLM then scores the target report against the eri- teria. Given the report, T, and P., it assigns a score Sc, E [0, 10] and justification for each sub- criterion c; C..
The final P-Score Sp is calculated by first computing each dimension score Sa as the weighted average of its sub-criteria. Then, Sp is obtained as the weighted average of the four dimension
5

---

Published as a conference paper at ICLR 2026
scores using dynamically generated weights. This can be formally expressed as:
Sp
Σ wa Sa
E wa
We
Sc,
dEDp
deDp
where Wa the dimension weight, We; is the sub-criterion weight, and se; is the sub-criterion score.
4.2 QUALITY CONTENT
Beyond personalization, we also assess the intrinsic quality of the generated report-its depth, in- sight, logic, clarity, and readability, regardless of user profiles. Quality is evaluated with respect to the task T and standards of rigorous research writing.
We define three dimensions: Depth & Insight (DEIN), Logical Coherence (LOGC), and Clarity & Readability (CLAR) (see Appendix E).
Evaluation Process. Following the dynamic criterion principle, an LLM meta-evaluator (i) as- signs weights (walde to the three dimensions, and (ii) generates a set of task-specific sub-criteria of for each dimension. A separate LLM scorer then rates the report against this criterion, produc- ing score E [0, 10] with justification for each sub-criterion E CJ. The final Q-Score is hierarchical weighted average:
SQ
Σ DED
W. Sa
Σ wi
Σ Wes
Scc
dEDo
where wa is the dimension weight, We, is the sub-criterion weight, and se, is the sub-criterion score.
4.3 R - FACTUAL RELIABILITY.
Although factuality metrics (Min et al., 2023; Wei et al., 2024; Chern al., 2023) exist, they are primarily designed to verify atomic facts against knowledge bases external search results. They remain unsuitable for deep research setting, where factuality must be evaluated through retrieved citations assess both the factual reliability of the report and the agent's capacity in utilizing web information. We therefore assess report reliability via an automated factual grounding framework, inspired by ResearcherBench (Xu et al., 2025) and DeepResearch Bench (Du et al., 2025). The process has three stages:
Claim Extraction and Deduplication. A Judge LLM employed to extract all verifiable factual claims with their sources, forming set of triplets TRI {(c, idx;, source, ) where uncited claims have empty sources (see Appendix J.3). A second pass deduplicates claims:
TRIunique Deduplicate
yielding Nottl unique claims, of which Neited are cited.
Automated Verification. For each unique triplet (c;,idx;, source;) E TTRI use the Jina Reader API to retrieve the source content Contenl,. Then the Judge LLM checks support:
1, if c, is supported by Content,, 0
(4)
0, if c; is unsupported or unknown.
Metric Calculation. We compute two key metrics from the verification results:
Factual Accuracy (FA) Measures the reliability of provided citations. It is the percentage of claims that are factually verified and supported by their corresponding source material.
Citation Coverage (CC) Assesses the proportion of factual claims in report that are supported by explicit citations, reflecting how well the content is evidence-bause
Finally, we average FA and CC derive single Factual Reliability score, SR:
FA
Σ
==
Nast
Us
10,
CC
Neted
x 10,
SR
FA + CC
(5)
N Weited
N

---

Published as a conference paper at ICLR 2026
4.4 FINAL SCORE AGGREGATION
To obtain a holistic measure of the report, we define the final overall score as a arithmetic mean over the three dimension scores:
Soverall Sp + SQ + Sr
(6)
3
where Sp, Sq. and SR denote the scores for personalization, quality and factual reliability respec- tively. This aggregation provides straightforward and comprehensive measure of the personalized deep research report.
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
We benchmarked a diverse set of systems, including commercial deep research systems: Gemini- 2.5-Pro Deep Research, O3 Deep Research, Perplexity Deep Research (Google DeepMind, 2025; OpenAl, 2025b; Perplexity Team, 2025), open-source deep research agents: Deerflow, Oagents, Miroflow (ByteDance, 2025a; Zhu al., 2025a; MiroMind AI Team, 2025), and leading LLMs 7enee-3.7-SSnne-SSeeee
Perplexity-Sone Saaeexity-Soaar-Raaoonng Reasoning- Pro, 1--r-ch-- Previeh (DeepMind; Anthropie; AI, 2025; OpenAI, a). Due to computational constraints, the evaluation was performed on subset of 150 representative queries. GPT-5 (OpenAI, 2025a) was utilized the judge model for Personalization (P) and Quality (Q) metries, while the more efficient GPT-5-Mini (OpenAI, b) served as the judge for the Reliability (R) metric, ensuring balance of advanced reasoning and efficiency (more details in Appendix and F).
with search tools: 2.5imii-..--ro-eeae
5.2 MAIN RESULTS
Table I: Evaluation results of Personalized Deep Research Bench under the Task w/Persona con- figuration. The best results in each column are highlighted in bold, and the second-best results are underlined.
Porsonalization
Quality
Reliability
Model
Overall
GOAL
CONT
PRES
ACTI
DEIN
LOGE
CLAR
EA
CC
Commercial Deep Research Agents
Gemini-2.5-Prr Deep Research
6.58
5.27
5.78
5.83
4.56
5.32
6.13
6,16
8.40
9.26
03 Deep Research Perplexity Deep Research
6.11
5.67
5,95
5.57
5.10
5.68
6,40
5.58
6.84
7.14
5.99
4,93
4.72
4.93
5.43
Open-Source Deep Revearch Agents
OAgents
6.64
6.68
6.44
7.13
6.92
6.99
7.44
6.85
3.77
8.32
Deerflow
4,97
5,43
6.85
2.32
MiroFlow
5.78
6.65
6.45
7.03
6.65
6.53
7.31
6.68
7.29
0.44
LLM with Search Tools
Gemini-2.5-PPp P-PPo w/Search
5.53
5.20
5.61
4.54
5.57
6.62
Claude 3.7 3.7-Sonnet w/Search
4,83
4.27
4.24
5.43
4.28
4.26
5.09
5.34
8.27
2.37
Perplexity-Son -onaart---Snnaaaaaee Reasoning.Pro
5.02
5.27
5.03
GPT-4.1 w/Search
4.28
4.59
4,86
5.74
4.07
4.21
5.27
5.54
6.75
0.10
The evaluation results on the Personalized Deep Research Bench under the Task w/Persona config- uration (Task and Persona are explicitly provided to the agent) are shown in Table 1. Our analysis reveals several key findings regarding the performance of different model categorics.
Open-source Agents Excel in Personalization. Open-source agents achieve the strongest per- sonalization, with OAgents achieving the top score (6.64) and leading most sub-metrics. including GOAL (6.68), PRES (7.13), and LOGC (7.44). MiroFlow also performs competitively, outperform- ing OAgents in CONT (6.45) and FA (7.29). However, reliability remains their weakness: OAgents suffers from low factual accuracy (3.77), while both MiroFlow and DeerFlow show poor citation coverage.
Commercial Agents Provide Balanced Quality and Reliability. Commercial systems achieve slightly lower personalization but higher reliability and consistent quality. Gemini-2.5 2.5-Pro Deep Re- search leads this group (6.58), achieving top FA (8.40), CC (9.26), and solid quality scores (DEIN:

---

Published as a conference paper at ICLR 2026
Perplexity-Sona--eessnnnnnn
MiroFlow
P-Score
Scores
PRES
CONT
P-Score
GOAL
CONT
PRES
ACTI
WiContext
wiPersone
Figure 3: Analysis on Personalization Metrics for Sonar- Reaao-nn Pro and MiroFlow
5.32, LOGC: 6.13, CLAR: 6.16). 03 Deep Research follows closely (6.11), leading in personaliza- tion within this category (GOAL: 5.67, CONT: 5.95) and maintaining competitive quality (DEIN: 5.68, LOGC: 6.40, CLAR: 5.58). In summary, commercial agents are reliable and robust in quality, but they lag moderately behind open-source agents in personalization.
LLMs with Search Tools Fall Short. Search Tools equipped LLMs underperform specialized agents. Gemini-2.5-Pro w/Search is the strongest in this group (5.53), while others, such as SomplexttySSnar-Reaaonnn
achieve high FA (8.44) but poor CC and weak personalization. GPT-4.1 w/Search, for example, nearly fails in CC (0.10). These results indicate that adding search alone is insufficient reach the personalization and quality of dedicated deep research agents.
5.3 IMPACT OF INFORMATION AVAILABILITY ON PERSONALIZATION
While the previous section evaluated agents with explicit personas, more realistic scenario involves inferring user needs from conversational or interaction context, as explicit personas are rarely avail- able. To examine this, conducted a comparative experiment under three conditions: Task Only (the agent receives only the task), Task w/Context (the task plus user's conversational or interaction context), and Task w/Persona (the task plus an explicit user persona, consistent with our main exper- iment). We present the results in Table and Figure 3. A more detailed results are in Appendix G.
Table 2: Evaluation results the Personalization across different personalization settings, The table shows results for three configurations: Task Only, Task w/Context, and Task w/Persona. Best scores in each column are highlighted in bold, second-best in underlined.
Personalization
Model
Setting
P-Score
GOAL
CONT
PRES
ACTI
Task Only
6.17
5.91
5,42
6.90
6.51
OAgents
Tusk w/Contett
6.53
6.32
5,99
7.04
Task w/Persona
6.78
6.68
6.44
7.13
6.92
Task Only
5.13
5.14
5.08
5.62
5.03
03 Deep Research
Task w/Context
5.48
5.58
5.67
3.70
5.29
Task w/Persona
5.46
5.67
5.95
5.57
Task Only
3.86
5.53
Gemini-2.5-Pro w/Search
Task w/Conten
4.55
4.66
4,95
5,59
4.09
Task w/Persona
4.70
4.85
3,20
3.61
4.19
More Information Consistently Yields Better Personalization. Across all systems, personaliza- tion scores (P-Score) increase with the user information (persona or context) provided. This trend holds for all sub-metrics, confirming the intuitive hypothesis that access to user-specific data is cru- cial for tailoring research outputs.
Explicit Personas Outperform Context. Context improves performance over the baseline, but the largest gains come from explicit personas. For instance, OAgents' GOAL score increases from 6.32 (Context) 6.68 (Persona), larger jump than the improvement from Task Only to Context.
8

---

Published as a conference paper at ICLR 2026
This indicates that while agents can partially leverage implicit context, they struggle to fully extract user preferences from unstructured, implicit data. Explicit personas, by contrast, provide a stronger and more accessible personalization signal.
5.4 BOOSTING PERSONALIZATION VIA CONTEXT-AWARE MEMORY SYSTEMS
The preceding analysis reveals that while contextual information is beneficial, agents struggle to dis- till into an actionable user understanding as effectively as when provided with an explicit persona.
To address this, we designed a second experiment to test whether advanced memory systems can transform unstructured context into explicit persona to perform personalized deep research. we evaluated 50 suitable queries three systems: Memo (Chhikara et al., 2025), Memory OS (Kang et al., 2025), and O-Mem (Wang et al., 2025b), on their ability to extract, integrate, and infer user preferences from context to drive downstream deep research systems. Results indicate potential for improving higher-level reasoning and user information integration.
Table 3: Evaluation results on for different memory systems under the Task w/Context setting, using Perplexity Deep Research. Currently, most memory systems can only align content with user char- acteristics, so we prioritize GOAL and CONT scores. We also display other metrics for clarity. The best metric highlighted in bold. Underlined denotes the second highest.
Personalization
Method
P-Score
GOAL
CONT
PRES
ACTI
No Mcmory
3.69
3.88
3.74
3.90
3.46
Mcm0
3,55
3.73
3.55
3.77
3.36
Memory OS
3.88
4.06
3.97
4.09
3.66
O-Mem
4.26
4.47
4.43
4.34
4.00
Task w/Persona
4.58
4.69
4.93
4.72
4.33
As shown in Table 3, memory systems yield varied but promising results. O-Mem outperforms both No Memory baseline and other systems, while Memo underperforms. However, significant gap remains between the best system and ideal Task w/Persona performance, indicating that current memory systems struggle to fully synthesize information from context. This gap highlights the need for future research on memory systems that combine factual retrieval with higher-level reasoning and abstraction, moving beyond storage toward constructing dynamic, persona-like models of users.
5.5 ALIGNMENT WITH HUMAN CONSISTENCY
To validate the evaluation framework, conducted systematic study comparing the judgments of LLMs against human experts. We sampled 15 representative queries and generated responses from two deep research agents: MiroFlow and O3 Deep Research. A panel of human evaluators scored these reports using the same criteria, establishing a ground truth for our comparison.
We designed complementary metrics quantify this alignment: Pairwise Comparison Agree- ment (PCA) measures the percentage of the LLM judge and human experts agree on which of the two reports better for specific criterion. Mean Absolute Rating Deviation (MARD) measures the average absolute difference between the scores assigned by the LLM and human judges. Detailed mathematical formulations for these metrics be found in Appendix C.
Based on the results shown in Table 4, GPT-5 achieved the highest PCA and lowest MARD, indi- cating the strongest agreement with human judgments, while maintaining a reasonable cost ($0.68 per query), We finally select GPT-5 5 as our primary judge model.
Table 4: Alignment results of judge LLMs with human ratings. PCA is reported proportion of agreement (higher is better), MARD as mean absolute deviation (lower is better), Avg. Cost is measured in US dollars (S). The best metric is highlighted in bold.
Judge LLM
PCA
MARD 1
Avg. Cost ($) 1
GPT-5
0.43
1.40
0.68
Claude-3.7- 3.7--onneet
0.39
1.44
0.97
Gemini-2.5 2.5-Pro
0.40
2.33
0.61

---

Published as a conference paper at ICLR 2026
6 CONCLUSION
In conclusion, work addresses the critical gap in DRAs evaluation by introducing the Personal- ized Deep Research Bench, the first benchmark of its kind featuring 250 realistic queries that pair 50 diverse deep research tasks across 10 domains with 25 authentic user profiles. Along with the PQR Evaluation Framework, which jointly measures personalization, content quality, and factual reliabil- ity, study reveals both the potential and current limitations towards personalized deep research. By establishing this rigorous foundation, our work paves the way for developing and benchmarking the next generation of truly personalized and effective AI research assistants.
7 ETHICS STATEMENT
This work strictly adheres to the ICLR Code of Ethics. The study protocol underwent internal compliance, privacy, safety review and obtained ethical approval within our institution. The research was conducted in accordance with applicable institutional and national research ethics regulations. User profiles were collected from volunteers under informed consent. Participants were informed of the research purpose, data usage, anonymization procedures, and public release format, and could withdraw any time without consequence. All data were strictly anonymized prior to release. The publicly released benchmark contains only de-identified and abstracted derivatives of the original profiles; personally identifiable information (PII) is included. Annotators and human evaluators participated voluntarily under informed consent and were fairly compensated. They were instructed ensure accuracy, neutrality, and confidentiality in their annotations and evaluations. To promote fairness and mitigate potential biases, tasks and profiles were designed to cover diverse domains, age ranges, professions, income levels, and life stages. The study does not involve sensitive or harmful content and was conducted in a controlled and ethically responsible manner.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide comprehensive details across the paper and appendix. Sec- tion 3 and Appendix describes benchmark construction, including task design, user profiles collec- tion, and query pairing. The evaluation framework is detailed in Section 4, including the dynamic weight allocation, granular criterion generation, and scoring methodology for each dimension. Sec- tion 5 and Appendix B outline experimental setups, systems, and configurations. Prompt templates (Appendix J). persona schema (Appendix D), and evaluation dimensions (Appendix E) are fully documented. We have submitted our evaluation data in the Supplementary Material. Due to Open- Review's file size limit, we only upload a subset of them. We will fully release the benchmark immediately after the double blind review process. We invite the community to build upon this work in advancing personalized deep research agents.
ACKNOWLEDGMENTS
We would like to express our sincere gratitude the anonymous reviewers for their thoughtful and constructive feedback. This work was supported by the National Natural Science Foundation of China (No. 62576307, No. NSFCU23B2055, No. NSFCU11B2027), the Fundamental Research Funds for the Central Universities (226-202--00138), Yongjiang Talent Introduction Programme 2021A-)56-G), and Information Technology Center and State Key Lab of CAD&CG, Zhejiang University.
REFERENCES
Perplexity AI. Sonar reasoning pro model (via perplexity api). Perplexity AI, 2025. URL https: //www, pe releppeee exity. bay...l. ai httphbbbblog/////ee/eee///ggoolt innto/iitooooiiiii the sonar paraa-proo-- api. Ac- cessed: 2025-09-10.
Anthropic. Claude 3.7 sonnet and claude code. URL UBR/sttpee//w..aatttpee.......o al claude-33 3- 7-sonnet. Accessed: 2025-09-10.
10

---

Published as a conference paper at ICLR 2026
ByteDance. Deerflow: A community deep research framework. https:: com/ bytedance/ / deer- flow, 2025a. Accessed: 2025-09-10.
ByteDance. Doubao deep research. https: www. wwwbaa.com/caaa....ooooooo.t. 2025b. Accessed: 2025-09-10.
Kaiyuan Chen, Yixin Ren, Yang Liu, Xiaobo Hu, Haotong Tian, Tianbao Xie, Fangfu Liu, Haoye Zhang, Hongzhang Liu, Yuan Gong, Chen Sun, Han Hou, Hui Yang, James Pan, Jianan Lou, Jiayi Mao, Jizheng Liu, Jimpeng Li, Kangyi Liu, Kenkun Liu, Rui Wang, Run Li, Tong Niu, Wen- long Zhang, Wenqi Yan, Xuanzheng Wang. Yuchen Zhang, Yi-Hsin Hung, Yuan Jiang. Zexuan Liu, Zihan Yin, Zijian Ma, and Zhiwen Mo. xbench: Tracking agents productivity scaling with profession-aligned real-world evaluations, 2025a. URL https : / /aaxxvivvv./////////////////aavl org/ wabbs2500 13651.
Zijian Chen, Xueguang Ma, Shengyao Zhuang. Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Sahel Sharifymoghaddam, Yanxi Li, Haoran Hong, Xinyu Shi, Xuye Liu, Nandan Thakur, Crystina Zhang, Luyu Gao, Wenhu Chen, and Jimmy Lin. Browsecomp-plus A more fair and transparent evaluation benchmark of deep-research agent, 2025b. URL aattps://axxtt /aaxivivvv....////// org/ abs/ 2508. 06600.
I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, and Pengfei Liu. Factool: Factuality detection in generative ai tool augmented framework for multi-task and multi-domain scenarios, 2023. URL https:
/ http://aas223....353333
Prateek Chhikara, Dev Khant, Saket Aryan, Taranject Singh, and Deshraj Yadav. Mem0: Building production-reade ai agents with scalable long-term memory. arXiv preprint arXiv:2504. 19413, 2025.
João Coelho, Jingjie Ning, Jingyuan He, Kangrui Mao, Abhijay Paladugu, Pranav Setlur, Jiahe Jin, Jamie Callan, João Magalhães, Bruno Martins, and Chenyan Xiong. Deepresearchgym: A free, transparent, and reproducible evaluation sandbox for deep research, 2025. URL https: / lawxiii.orgaas///20.
19253.
Google DeepMind. Gemini 2.5 technical report and model card (referencing gemini 2.5 pro). URL https: / deepmind. google models s commii//pro//// Accessed: 2025-09-10.
Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. Deepresearch bench:
A comprehensive benchmark for deep research agents, 2025. URL https://arxiv.....
abs/2506 11763.
Gerhard Fischer. User modeling in human computer interaction. User modeling and usereadapted interaction, 11(1):65-86, 2001.
Google DeepMind. Introducing gemini deep research. https://gemini gowgle/ overview/ researchh, 2025. Accessed: 2025-09-10.
Boyu Gou, Zanming Huang. Yuting Ning, Yu Gu, Michael Lin, Weijian Qi. Andrei Kopanev, Botao Yu, Bemal Jiménez Gutiérrez, Yiheng Shu, Chan Hee Song. Jiaman Wu, Shijie Chen, Hanane Nour Moussa, Tianshu Zhang, Jian Xie, Yifei Li, Tianci Xue, Zeyi Liao, Kai Zhang, Boyuan Zheng, Zhaowei Cai, Viktor Rozgic, Morteza Ziyadi, Huan Sun, and Yu Su. Mind2web 2: Evaluating agentic search with agent-as-a-judge, 2025. URL https: //a //axiiiivv.. org/ / 2506. 21506.
Jian Guan, Junfei Wu, Jia-Nan Li, Chuanqi Cheng, and Wei Wu. A survey personalized alignment the missing piece for large language models in real-world applications, 2025. URL https: / //arxii.orr/////e abs 253.. 7003.
Mengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie, Bowei Xia, Tao Sun, Ziyu Ye, Zhaoxuan Jin, Yingru Li, Qiguang Chen, Zeyu Zhang, Yifeng Wang. Qianshuo Ye, Bermard Ghanem, Ping Luo, and Guohao Li. Owl: Optimized workforce learning for general agent assistance in real-world task automation, 2025. URL https: h/tp://aaavvvvv/// abb /250 23885.
11

---

Published as a conference paper at ICLR 2026
Yuxuan Huang, Yihang Chen, Haozheng Zhang, Kang Li, Huichi Zhou, Meng Fang, Linyi Yang, Xiaoguang Li, Lifeng Shang, Songcen Xu, Jianye Hao, Kun Shao, and Jun Wang. Deep research agents: A systematic examination and roadmap, 2025. URL https://azxiv. org./v.orraaaaoo abs/ 2506. 18096.
Bowen Jiang, Zhuoqun Hao, Young-Min Cho, Bryan Li, Yuan Yuan, Sihao Chen, Lyle Ungar, Camillo J. Taylor, and Dan Roth. Know me, respond to me: Benchmarking Ilms for dynamic user profiling and personalized responses at scale, 2025. URL htttps://axxiv.rrpg abs/ 2504. 14225.
Jiazheng Kang, Mingming Ji, Zhe Zhao, and Ting Bai. Memory os of ai agent, 2025. URL https: / //arxiv. orgg abs 2aba/255. 22006 06326 6.
Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott Hale. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Intellignce, 6(4):383-392, 2024.
Weizhen Li, Jianbo Lin, Zhuosong Jiang, Jingyi Cao, Xinpeng Liu, Jiayu Zhang, Zhenqiang Huang, Qianben Chen, Weichen Sun, Qiexiang Wang, et al. Chain-of-agents: End-to-end agent founda- tion models via aulti-agent distillation and agentic rl. arXiv preprint arXiv:2508. 13167, 2025a.
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models, 2025b. URL abss 22511.05666
https:
Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji- Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability, 2025c. URL https:://// petttts://ariiee www.ia.ivvvv..oo/..a org/ abs 2504 2170721776.
Manus AL. Leave it to manus. https::/mmamatii im/ 2025. Accessed: 2025-09-10.
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general assistants, 2023. URL https://arxiv.or
abs /2122111111 12983.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of fac- tual precision in long form text generation, 2023. URL attps::/arxivv org/ hbtg//bbb/255555 14251.
MiroMind AI Team. Miroflow: An open-source agentic framework for deep research. https:
//github. http./.....//....oo.....///////
2025.
Moonshot AI. Kimi-researcher: End-to-end training for emerging agentic capabilities. https: /tt///mmmieeeeaceee//./. 2025. 2025. Accessed: 2025-09-10.
/ //moonshotai. hww.:/shhttii...hbbbaoo..
OpenAl. Introducing GPT-4.1 1 in the api, a. WtRphhtts:///peeett http:///ppaaaa.ccmmmoooaaae www./ooow.////////co/ooooccowcce
Accessed: 2025-09-10.
OpenAl. Using GPT-5 OpenAl API fincludes GPT-5 mini model details], b. URL https: Accessed: 2025-09-10.
//ppatform..opeete openai coo/docs/////oooccoooccccccccccc
OpenAl. Introducing gpt-5, 2025a.
https://opena...cmmndeee
introducing
5/. 5/.
OpenAl. Introducing deep research.
https://openai.com//nce
introducinn--eee--eeeeenn 2025b. Accessed: 2025-09-10.
Perplexity Team. Introducing perplexity deep research. https:
-eppe research,,,,h, 2025. Accessed: 2025-09-
10.
Long Phan, Alice Gatti, Ziwen Han, et al. Humanity's 's last exam, 2025. URL org/abs/ /2011 14249.
12

---

Published as a conference paper at ICLR 2026
Omid Rafician and Hema Yoganarasimhan. Ai and personalization. Artificial intelligence in mar- keting, pp. 77-102, 2023.
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large lan- guage models meet personalization, 2024. URL https: :/ arxiv.
http://awwwww..a//////////////// www//bs///////oo.ooo.ooo. 11406.
Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpuro- hit, Ameet Deshpande, Karthik Narasimhan, and Vishvak Murahari. Personagym: Evaluating persona agents and Ims,, 2025. URL https:
18416.
Dingfeng Shi, Jingyi Cao, Qianben Chen, Weichen Sun, Weizhen Li, Hongxuan Lu, Fangchen Dong, Tianrui Qin, King Zhu, Minghao Liu, et al. Taskcraft: Automated generation of agentic tasks. arXiv preprint arXiv:2506. 10055, 2025.
Xiangru Tang, Tianrui Qin, Tianhao Peng, Ziyang Zhou, Daniel Shao, Tingting Du, Xinming Wei, Peng Xia, Fang Wu, He Zhu, et al. Agent kb: Leveraging cross-domain experience for agentic problem solving. arXiv preprint arXiv:2507.06229, 2025.
Meiling Tao, Chenghao Zhu, Dongyi Ding, Tiannan Wang, Yuchen Eleanor Jiang, and Wangchun- shu Zhou, Personafeedback: A large-scale human-annotated benchmark for personalization, 2025. URL https: org/ aba/aba//222 /2006 12915.
Danqing Wang, Kevin Yang. Hanlin Zhu, Xiaomeng Yang, Andrew Cohen, Lei Li, and Yuandong Tian. Learning personalized alignment for evaluating open-ended text generation, 2024a. URL https: /arxivv..vvvv org/ abss 2210. 03304.
Ningning Wang. Xavier Hu, Pai Liu, He Zhu, Yue Hou, Heyuan Huang, Shengyu Zhang, Jian Yang, Jiaheng Liu, Ge Zhang, Changwang Zhang, Jun Wang, Yuchen Elcanor Jiang, and Wangchunshu Zhou. Efficient agents: Building effective agents while reducing cost, 2025a. URL https: //arxiv.org./ abs 2508 02694.
Piaohong Wang. Motong Tian, Jiaxian Li, Yuan Liang, Yuqing Wang, Qianben Chen, Tiannan Wang, Zhicong Lu, Jiawei Ma, Yuchen Eleanor Jiang, and Wangchunshu Zhou. O-mem: Omni memory system for personalized, long horizon, self-evolving agents, 2025b. URL https://arxix: org/aba htgg/ab//ie abs/2bbs11111 /251111111 13593.
Tiannan Wang, Meiling Tao, Ruoyu Fang, Huilin Wang, Shuai Wang, Yuchen Eleanor Jiang, and Wangchunshu Zhou. Ai persona: Towards life-long personalization of llms, 2024b. URL
https://arxix: org/ aaas/////222 13103.
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents, 2025. /RLLhttps///rxxi.oorr wttpp/wwwww..../////////0// 12516.
Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoe V. Le. Long-form factuality in large language models, 2024. URL. https: http://wwwww.....//////////
18802.
xAI Team. Introducing grok deepsearch. https: http:/://xxa.//w///ge///// 2025-09-10.
2025. Accessed:
Tianze Xu, Pengrui Lu, Lyumanshan Ye, Xiangkun Hu, and Pengfei Liu. Researcherbench: Eval- uating deep ai research systems on the frontiers of scientific inquiry, 2025. URL https:
//arxiv..o warxiv.org/abe...
Zheng Zhao, Clara Vania, Subhradeep Kayal, Naila Khan, Shay B. Cohen, and Emine Yilmaz. Personalens: A benchmark for personalization evaluation in conversational ai assistants, 2025.
URL https: /tt:://axxii /tt.xviivvv..///// org/ abs///bs/2550 wttp://v.rrggb////....ooo 09902.
Wangchunshu Zhou, Yuchen Eleanor Jiang. Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang, Jing Chen, Ruipu Wu, Shuai Wang, Shiding Zhu, Jiyu Chen, Wentao Zhang, Xiangru Tang, Ningyu Zhang, Huajun Chen, Peng Cui, and Mrinmaya Sachan. Agents: An open-source framework for autonomous language agents. 2023. URL https: : / arxiv. wtgg/bb///////////// 07870.
13

---

Published as a conference paper at ICLR 2026
Wangchunshu Zhou, Yixin Ou, Shengwei Ding, Long Li, Jialong Wu, Tiannan Wang, Jiamin Chen, Shuai Wang, Xiaohua Xu, Ningyu Zhang. Huajun Chen, and Yuchen Eleanor Jiang. Sym- bolic learning enables self-evolving agents. 2024. URL 18532.
He Zhu, Tianrui Qin, King Zhu, Heyuan Huang, Yeyi Guan, Jinxiang Xia, Yi Yao, Hanhao Li, Ningning Wang, Pai Liu, Tianhao Peng, Xin Gui, Xiaowan Li, Yuhui Liu, Yuchen Eleanor Jiang, Jun Wang. Changwang Zhang, Xiangru Tang. Ge Zhang, Jian Yang, Minghao Liu, Xitong Gao, Jiaheng Liu, and Wangchunshu Zhou. Oagents: empirical study of building effective agents, 2025a. URL https: arxiv. org/ abs 2506. 15741.
King Zhu, Hanhao Li, Siwei Wu, Tianshun Xing, Dehua Ma. Xiangru Tang, Minghao Liu, Jian Yang, Jiaheng Liu, Yuchen Elcanor Jiang, et al. Scaling test-time compute for Ilm agents, 2025. URL https://rrxiv. org/abs/2506. /2928, 2025b.
Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. Personality alignment of large language has//ab//at /at/aaba///400 220..400...00 11779.
models, 2025c. URL
Thomas P. Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li, and Hongseok Namkoong. Per- sonalllm: Tailoring Ilms to individual preferences, 2025. URL https: paapps:///rxxiee /arxiv.ivvv/.////////aa////aaa wttpiiss.rg////////// 200962
14

---

Published as a conference paper at ICLR 2026
A LIMITATIONS
This work still has some limitations that must be admited: a) The collection of user personas and context annotations was conducted in Chinese. Although a parallel English version was provided, the underlying content remains linguistically and culturally constrained. b) Due to computational constraints, our main experiments were conducted on a selected subset of queries rather than the full benchmark. For future work, we plan to expand persona construction and context annotation beyond the Chinese-centric setting, aiming for more diverse and cross-lingual coverage. We also intend to scale up experimental scope fully utilize all benchmark queries and explore richer evaluation protocols under varied computational settings.
B EXPERIMENT DETAIL.
B. CONHIGURATION METHODS
To ensure fair comparison among different agents, standardized their execution budgets and search configurations. Specifically, open-source deep research agents: Deerflow, Oagents, Miroflow all standardized on GPT-S-Mini as the base LLM. The maximum number of execution steps was set to 8 for OAgent. For Deerflow: max_ Step_num num and max max_mlan___aaapppppp aalan_iteraa www.iiteaaioon ons are default set to and 1. To Miroflow: max max_turns turns and ma max_tool tool calls _ar___ee__________ purr in main mmin__geea agent and sub _aagenttsss are both default set to 20 and 10. All agents relied on SerperAPI for web search and Jina for web content retrieval. For agents equipped with built-in web search tools, we set the reasoning_efferteeeeeeeeeeeeeeee _ffrt parameter to medium. In addition, for sonar-reasoning-pro the sea rearchhccntext. _iieeitttiiieeee parameter was also set to medium.
B.2 DATA SELECTION
Given the high cost of rumning our full evaluation pipeline, we first reduced the original set of 250 queries. These queries covered 50 distinct tasks, each originally paired with 5 personas. To make the evaluation more tractable, we limited each task to 3 representative personas, resulting in a reduced set of 150 queries. From this subset, we further selected 50 queries that reflect broad range of personalization demands across different user goals and characteristics. These queries were chosen based on their potential to reveal how effectively memory system can adapt its responses to individual users. Since current memory systems primarily limited to aligning content with user profiles rather than deeper task-level adaptation, our evaluation emphasizes Goal Alignment and Content Alignment the core metrics. Additional metries also reported for completeness and transparency.
C FORMULATION OF HUMAN CONSISTENCY METRICS
This section provides the detailed mathematical definitions for the metrics used to evaluate the align- ment between LLM judges and human experts.
Pairwise Comparison Agreement (PCA). For each query q and criterion c, let A and B denote the two reports being compared. We denote the model's scores as mye and m and the human ground truth scores as h4 and h PCA is defined as the proportion of cases where the model's preference order matches the humans' preference order:
1 PCA
Σ 2.c
[s m..) sgn(hi) n1te)
N
where N is the total number of query-criterion pairs, sgn(æ)) is the sign function, and 1[:] is the indicator function, which equals if the condition true and otherwise.
Mean Absolute Rating Deviation (MARD). For each query q, report r E {A, B). and criterion the MARD is the overall mean of the absolute deviations between the model scores (m, and
15

---

Published as a conference paper at ICLR 2026
the human scores (Ky,e). It is calculated as:
MARD
1 E,
E
Σ
Im
rE(A,B) CEC
where [Cyl is the number of criteria for query q, and the term 2/C₂1 accounts for the two reports being evaluated for each criterion.
D PERSONA SCHEMA
Table 5: Predefined Schema For Persona Collection (All Private Data Anonymized for Release)
Predefined Persona Schema.
Basic Attributes
Identity Characteristics
Name, Age, Gender, Occupation Family Members and Relationships, Pets Permanent Residence, Hometown
Family Status
Long-term Spatial Characteristies
Behavioral Characteristics
Online Usage Habits
High-frequengy Apps and Usage Duration, Online Social Be- havior
Ofline Long-term Behavior
Daily Routine, Consumption Cycle, Consumption Characteris- tics, Periodic Mobility
Environment
Time
Time Preference
Geographical Location
Frequent Places, Travel Radius
Personality Traits
Personality, Decision Docision-making Style, Shopping Preference
Preferences and Interests
Lifestyle Preferences
Diet, Accommodation Preferences, Shopping, Services
Travel Preferences
Frequency, Destinations, Travel Style
Content Preferences
Article Collection, Short Videos, Screenshots, Books, Movies, Singers, Actors
Exercise Preferences
Exercise Habits, Exercise Goals, Exercise Types, Other Invest- ments, Exercise Locations
Health Status
Physical Condition
Medical History, Mental Condition, Physical Fitness
Health Needs
Health Needs
Financial Information
Financial Status
Income Structure, Asset Status, Consumption Characteristics, Debt Situation
Investment Experience
Investment Background, Knowledge Level
Risk Management
Risk Appetite
16

---

Published as a conference paper at ICLR 2026
E
DEFINITIONS OF EVALUATION DIMENSIONS
Table 6: Dimensions and Definitions For Personalization Evaluation.
Dimension
Description
Goal Alignment
How well the report addresses the user's explicit and implicit goals.
Content Alignment
The suitability of the report's topic, depth, and breadth for the user's ' knowledge and interests.
Presentation Fit
The alignment of the report's 's language, structure, and style with the user's s comprchension and preferences.
Actionability&Praciiiie
The extent to which the report offers practical value for decision- making or action.
Table 7: Dimensions and Definitions For Quality Evaluation.
Dimension
Description
Depth & Insight
The analytical richness, originality of thought, and critical per- spective exhibited by the report.
Logical Coherence
The logic and coherence of the report's reasoning, ensuring ideas are rigorous and easy follow.
Clarity & Readability
The report's 's language, information presentation, and formatting.
F
RUNNING COST
Table 8: Running Cost Comparison Across Models and Personalization Settings (per query)
/odel/ Setting
Task Only
Task w/ Persona
Task w/ Context
Open-Source DRAs (GPT-5-Mini based)
OAgents
$1.60
$1.70
$3.00
DeerFlow
$0.50
$0.57
$1.20
MiroFlow
$1.00
$1.11
$2.10
LLM with Search Tools
Gemini-2.5-PP w/ Search
$0.05
$0.06
$0.24
Claude-3.7 3.7-onnet w/ Search
$0.03
$0.04
$0.31
Perplexity-Sone
Sapplexit--Sonr-PPPe Sonar-Reasoning
$0.02
$0.03
$0.78
GPT-4.1 w/ Search
$0.02
$0.02
$0.31
17

---

Published as a conference paper at ICLR 2026
G A MORE DETAILED EXPERIMENTS RESULTS
Table Evaluation results on the Personalization across different personalization settings to vari- agents. The table shows results for three configurations: Task Only, Task w/Context, and Task w/Persona. Best scores in each column are highlighted in bold, second- best in underlined.
Model
Setting
P-Score
GOAL
CONT
PRES
ACTI
Commercial DeepResearch Agents
Task Only
4.57
4.12
3.90
6.49
4.77
Gemini-2.5-Pro
w/Context
4.70
4.84
5.17
5.66
4.18
w/Persona
5.12
5.27
5.78
5.83
4.56
Task Only
5.13
5.14
5.08
5.62
5.03
03
w/Context
5.48
5.58
5.67
5.70
5.29
w/Persona
5.46
5.67
5.95
5.57
5.10
Task Only
3.58
3.47
3.38
4.82
3.47
Perplexity
w/Context
4.19
4.23
4.29
4.56
4.06
w/Persona
4.58
4.69
4,93
4.72
4.33
Open-Source DeepResearch Agents
Task Only
6.17
5.91
5.42
6.90
6.51
OAgents
w/Context
6.53
6.32
5.99
7.04
6.81
w/Persona
6.78
6.68
6,44
7.13
6.92
Task Only
5.11
4.77
4,41
6.67
5.31
Deerflow
w/Context
5.02
4.77
4.47
6.60
5,09
w/Persona
5.38
5.20
4.97
6.71
5.41
Task Only
5.82
5,51
5,09
6.78
6.15
Miroflow
w/Context
6.07
5.93
5.50
6.76
6.26
w/Persona
6.65
6.65
6.45
7.03
6.65
LLM with Search Tools
Task Only
3.96
3.91
3,86
5.53
3.70
Gemini-2.5-PP
w/Context
4.55
4.66
4.95
5.59
4.09
w/Persona
4.70
4.85
5.20
5.61
4.19
Task Only
4.00
3.83
3.63
5.32
3.99
Claude-3.7- 1.ee-3.7-Snnee Sonnet
w/Context
3.85
3.87
4.06
4.80
3.54
w/Persona
4.37
4.27
4.24
5,43
4.28
Task Only
3.52
3.44
3.30
4.80
3.40
Sonar-Rca-Pro
w/Context
4.15
4.19
4.18
5.21
3.98
w/Persona
4.27
4.27
4.37
5.27
4.15
Task Only
3.79
3.71
3.63
5.44
3.55
GPT4.1
w/Context
4.41
4.43
4.52
5.70
4.08
w/Persona
4.52
4.59
4.86
5.74
4.07
18

---

Published as a conference paper at ICLR 2026
H
CASE STUDY
Deep Research Task
am beginner runner preparing for first full marathon in the next three manths, with the goal of completing the race safely and without injury. can train about five days week for roughly one hour per session, mainly running nearby parks residential roads.
do not have professional running equipment-onky pair of basic running shoes. currently have no running experience and no specific pace targets, hope to gradually increase my mileage while considering my daily life
schedule.
The training plan should be to follow, sustainable, and adaptable. Please create personalized training plan based on these conditions
Personal Description
Personal Description
Male, 34-year-old, Supply chain management professional
Female graduate student, Clinical paychology specialization
Minimal exercise engagement, High personal demands
Three weekly exercise sessions, Yoga and light jogging preferences, based and proximate outdoor exercise
Married status, 6-year-old daughter, Alaskan Malamute ownership
00oo
Morning chronolype orientation, Early morning study and writing optimization
Mild obesily condition, Nock discomfort symptoms, Overall health improvement aspirations
Mild obesily condition, Neck discomfort symptoms, Overall health improvement aspirations
B
High-demand professional role, Frequent business travel requirements
Scholarship dependency, Part-time employment supplementary Income
Personalized Report
Personalized Report
1. Designate weekends as family fitness integration periods, incorporating Alaskan Malamnue into running activities for joint
Integrate running as mindfulness-based practice utilizing psychological training for enhanced self-observation and emotional regulation.
family-dog exercise engagement.
Optimize moming chronolype establishing running as the primary
Develop gradual weight reduction
protocols with specialized running posture
moming routine for maximum cognitive and physical benefits.
guidance prevent discomfort
3. Facilitate peer interaction through universily or community running club participation and social media training documentation.
exacerbation
3. Implement business travel-adapled training solutions utilizing hotel fitness facilities with progressive beginner- orientad training progression plans.
Implement cost-effective equipment complimentary training and mobile application recommendations minimize financial barriers.
Figure 4: Case Study of Personalized Deep Research
19

---

Published as a conference paper at ICLR 2026
I DATA ANNOTATION
Our context dataset was manually annotated by 6 trained annotators, resulting in 5,939 labeled instances, The process required a total effort of 85 person-days and a budget of approximately $6,000 USD.
Data Annotation Protocol
Task Objective
The core of this task is to simulate the behavior of specific user persona by collecting various types of information that this user would likely encounter, consume, or generate in their daily life. The final goal is to create high-quality memory database for each persona that accurately reflects their unique characteristics.
Core Quality Standard: Reversibility
This is the most critical quality standard for this task. Every piece data you collect must clearly point to specific trait of the user persona.
Verification Method:
After collection, review all data and assess whether its con- tent is sufficient to reverse-infer the user's key persona traits, such their profes- sion, interests, and personality.
Acceptance Criteria: All collected data must pass the reversibility test be con- sidered high-quality. If the data is too generic and the persona cannot be inferred from the data will be deemed non-compliant.
Collection Based on Persona Preferences
Carefully read every preference tag the user persona description.
For each preference, collect at least five pieces of related content.
Ensure Diversity of Sources and Types
Source Diversity: Do not limit collection to a single platform. Data can be gath-
ered from various apps (e.g., Twitter, Instagram, Reddit), websites, forums, etc.
Type Diversity: Diversify the format of the data you collect. Examples include:
Screenshots of social media posts.
Screenshots of conversations with friends (that reflect opinions or prefer- ences).
Links and titles of articles, news, or videos.
Screenshots of purchase histories or product reviews.
Requirement for Conversational Content
Between 20% and 50% of the total data collected for each user should be in conversational format. This helps more vividly showcase the user's personality and communication habits.
Add Reasonable Noise for Authenticity
The content you collect does not need to be an exact match to the persona's description. You can add relevant and reasonable details or noise to make the data appear more authentic, as if generated by real user.
Example: the persona likes basketball, you could collect a news article about recent Lakers game or screenshot of conversation with friend debating whether Jordan or LeBron the better player.
Note: Any added noise must not contradict other defined attributes in the persona, such as spending habits, personality, or profession.
Quantity Quality
There are no strict quantitative requirements for data collection. Please prioritize quality and collect much rich data possible. Quality always takes precedence over quantity.
Deliverables and Annotation Requirements
Deliverable: For each user persona, export separate a aimemory database file.
Content Annotation: During collection, each piece of content must be given clear title and be correctly associated with its corresponding persona.
20

---

Published as a conference paper at ICLR 2026
J PROMPT TEMPLATES
J.1 PROMPTS IN PERSONALIZATION EVALUATION
Prompt for Personalization Dimension Weights Allocation
You are experienced evaluation expert for research articles. You excel at deeply un- derstanding goals, challenges, and key value points of a specific research task and the task initiator's s persona, and then setting dynamic, reasonable, and well rell-justified weights for evaluation dimensions in subsequent personalized article assessments. <lsystem_roles
<user_ pnser_prompt>
Here is deep research task, follows:
<task>
[taak__rompp)
</task>
The user persona is follows:
<persona>
persona_proopp|""
</persona>
<instruction>
Background: The research team will conduct in-depth and comprehensive research based on the above <task> and <persona> and ultimately produce high-quality, personalized research article.
Your task: As the evaluation expert, you need the weights of the personalized evalua- tion criteria for this specific <task>, The evaluation will revolve around the following four dimensions:
Whether the research sufficiently and accurately understands the relationship between the task and the user persona, extracts deep and implicit needs, and generates a personalized report based on them.
Goal Alignment:
2. Content Alignment: Whether the research selects and customizes content accord- ing to the user's interests, knowledge background, and preferences.
3. Actionability & Practicality: Whether the report is feasible, practical, and helpful for the user's decision-making.
4. Presentation Fit: Whether the report's language style, information structure, and presentation format match the user's cognitive habits and medium preferences.
Evaluation formula:
Total score = Goal Alignment Goal Aligmment weight + Content Alignment Content Alignment weight Actionability & Practicality * Actionability Practicality weight Presentation Fit Presentation Fit weight. (Note: The sum of all weights must be exactly
1.0)
Core requirements:
1. Deeply analyze the task and user persona: Carefully study the specific content of <task>, its explicit goals, potential challenges, and hidden objectives. Combine this with <persona> to analyze the user's needs, background, and preferences, and understand the core value of the task's outcome.
2. Dynamically assign weights: Based on your analysis, assign weights to the four dimensions (use decimals between 0 and 1, e.g., 0.30). The key is to recognize that different tasks and personas emphasize different aspects, so weights must be flexibly adjusted according to task characteristics and persona, not fixed.
3. Explain your reasoning: Your analysis (canalysis>) must clearly and specifically explain why each dimension assigned a given weight, and directly link your
21

---

Published as a conference paper at ICLR 2026
reasoning the requirements of <task> and the characteristics of <persona>, This is critical for evaluating the quality of your work.
4. Output in the standard format: Strictly follow the example format be- low: first output <analysis> with detailed reasoning, then immediately provide <json_ ousou_output> with the weight assignment results.
</instruction>
<examples rraiieaalstionne>>>>
Below are two examples that demonstrate how to adjust the evaluation dimension weights and explain the reasoning based changes in task nature and user persona. Focus on learning the thinking process and analytical method in these examples, not simply copying their content weight values.
</examples_ rationall>
Now strictly follow the above instructions and methodology, and start your work for the following task:
<task>
"[ ([task prompt)"
</task>
<persona>
peeronna_prompt]"
</persona>
Please output your <analysis> and <json_outputs output>.
Prompt for Goal Alignment Criteria Generation
You are an experienced research article evaluation expert. You excel at breaking down ab- stract evaluation dimensions (such "Goal Understanding and Personalization Insight") into actionable, clear evaluation criteria tailored to the specific research task and user per- sona, and assigning reasonable weights with explanations for each criterion. </ </system_rooe role>
<user pere_ppropp>>>>
Background: We are evaluating a research article written for the following research task under the dimension of Goal Alignment.
Goal Alignment: Whether the research fully and accurately understands the relationship between the task and the user persona, extracts deep and implicit needs, and generates personalized report based on that understanding, with focus performing user-centered, deeply personalized matching between the user persona and task requirements.
<task>
(ttask_prompp
</task>
The user persona is follows:
<persona>
"
</persona>
<instruction
Your goal:
For the Goal Alignment dimension of this research article, formulate set of detailed, spe- eific, and highly targeted evaluation criteria that are tightly aligned with the above <task> and <persona>. You need to:
1. Deeply analyze the user persona and task scenario: Thoroughly examine the back- ground characteristics, knowledge structure, cognitive habits, and latent expec-
22

---

Published as a conference paper at ICLR 2026
tations of <persona>. Combine this with the specific application scenario of <task> to identify the user's 8 core explicit needs and deeper implicit needs.
2. Formulate personalized evaluation criteria: Based on the above analysis, propose specific evaluation criteria that reflect deep understanding of <persona> and close fit to the <task> scenario. These criteria should assess whether the content is well adapted the user persona in style, depth, perspective, and practicality.
3. Explain the personalization rationale: Provide brief explanation (explanation) for each criterion, clarifying how addresses the specific attributes <persona> or special requirements of <task>, and why such targeting critical achieving good match.
4. Assign rational weights: Assign weight (weight) to each criterion, ensuring that the total sum is 1.0. The distribution of weights should directly reflect the rela- tive importance each criterion measuring how well the content matches "this particular user" in "this particular task." The closer criterion is tied to persona characteristics and task scenario, the higher its weight should be.
Core requirements:
1. Deep personalization orientation: The analysis. criteria, explanations, and weights must be deeply rooted in the uniqueness of <persona> (c.g., their professional background, cognitive level, decision-making preferences, emotional needs) and the specific context of <task>, Avoid generic or templated evaluation.
2. Focus on contextual responsiveness and resonance: The criteria should evaluate whether the content not only responds to the task the informational level but also resonates with the context and expectations implied by the user persona in terms of expression style, reasoning logic, case selection, and level of detail.
3. Rationale must reflect targeting: The <ana lysiss section must clearly explain how key features were extracted from the given <persona> and <task> to form these personalized criteria. Each criterion's explanation must directly show how serves this specific user and task.
4. Weights must reflect personalization priorities: The weight distribution must logi- cally demonstrate which aspects of alignment the most critical success factors for "this user" completing "this task."
5. Standard output format: Strictly follow the example format below. First output the <analysis> text, then immediately provide the iison_output:>
</instruction>
<example_ratioraal)
The example below demonstrates how develop Goal Alignment evaluation criteria based on the task requirements. Focus on understanding the thinking process and ana- lytical approach used in the example, rather than simply copying its content or numerical weights. //example_ratione
Please strictly follow the above instructions and methodology. Now, for the following spe- cific task, start your work:
<task>
"(task_prompl) prompt)"
</task>
<persona>
"ppersona_proopp]"
</persona>
Please output your <analysis> and <json_output>, </user_prompt>
23

---

Published as a conference paper at ICLR 2026
Prompt for Content Alignment Criteria Generation
You are an experienced research article evaluation expert. You are skilled breaking down abstract evaluation dimensions (such as "Content Alignment") into actionable, clear, and specific evaluation criteria tailored the given research task and user persona, and assigning reasonable weights and explanations for each criterion. </system_
<user peeer__pompp>
Background: We are providing a personalized scoring rubric for a specific task and user persona from the dimension of Content Alignment. Content Alignment: Whether the research content customized based on the user's inter- ests, knowledge background, and other preferences.
<task>
"[task_ pussppprompt)"
</task>
The user persona is as follows:
<persona>
"[ "(persona_prompp"""
</persona>
<instruction>
Your Goal: For the Content Aligmment dimension of this research article, create set of de- tailed, concrete, and highly tailored evaluation criteria for the above <task> and <persona>. You need to:
Analyze the Task and Persona: Deeply analyze <task> and <persona> to infer the user's potential interests, knowledge background, and the depth and breadth of content they may prefer.
2. Formulate Criteria: Based on your analysis, propose specific evaluation criteria that focus on whether the report's content matches the user's interest points and knowledge level.
3. Provide Explanations:
For each criterion, provide brief explanation (explanation) explaining why it is important for evaluating the content align- ment for this <task>.
4. Assign Weights: Assign reasonable weight each criterion (weight), ensur- ing that the sum of all weights equals exactly 1.0. The weight allocation should logically reflect the personalization-fits principle: criteria directly tied unique personal traits, exclusive preferences, or specific contextual needs in the user per- sona should receive higher weights, they are key achieving true personalized content alignment.
5. Avoid Overlap: Make sure the evaluation criteria focus solely on the Content Alignment dimension, avoiding overlap with other dimensions such as Goal Align- ment, Expression Style Alignment, and Practicaity/AAtiionbiity
Core Requirements:
Strongly Linked the Persona:
The analysis, criteria, explanations, and weights must be directly connected to the user's interests, knowledge background, con- tent preferences.
2. Focus on Content Selection and Depth:
The criteria should assess whether the choice of content is precise and whether the depth is appropriate, rather than merely evaluating whether information presented.
3. Provide Sufficient Rationale: The <analysis> section clearly articulate the overall reasoning behind formulating these criteria and weights, linking them to <task> and <persona>. Each explanate on must clarify why the individual cri- terion is relevant.
24

---

Published as a conference paper at ICLR 2026
4. Reasonable Weighting: The weight distribution should be logical, reflecting the relative importance of each criterion in measuring content aligmment, with particu- lar emphasis on giving higher priority to personalized aspects.
5. Standardized Output Format: Strietly follow the format below output the <analysis> text first, immediately followed by <json_output>.
</instruction>
<example_ eexample_rational>>>>>>>>>>>>>>> rational>
The following example demonstrates how to formulate content alignment evaluation cri- teria based on the task requirements and user persona. Pay close attention to the think- ing process and analytical approach in this example, rather than simply copying the con- tent weight values. </example rational>
Please strictly follow the above instructions and methodology. Now, for the following spe- eific task, start your work:
<task>
(tass_prompt]
</task>
<persona>
[[persona "artoona_pomp""""""
</persona>
Please output your <analysis> and <json_oune jison_outpub>... <luser_prompt>
Scoring Prompt for Personalization
<system, Baciinnooeeeeeie You are strict, meticulous, and objective expert in evaluating personalized research articles. You excel deeply evaluating research articles based specifie personal- ization assessment criteria, providing precise scores and clear custifications.<<ssttee
<user prompt>
Task Background
You given in-depth research task. Your job is evaluate research article written for this task in terms of its performance in "Personalization Alignment". We will evaluate it across the following four dimensions:
1. Goal Alignment
2. Content Alignment
3. Presentation Fit
4. Actionability & Practicality
<task>
[tass_pprompt"
</task>
User Persona
<persona>
(persona_prompt]"
</persona>
Article to be Evaluated
Barget_articles
"(article)" </target_artccl>>
Evaluation Criteria
25

---

Published as a conference paper at ICLR 2026
You evaluate the specific performance of this article in terms of personalization align- ment, following the criteria list below, outputting your analysis and then assigning a score from 0-10. Each criterion includes its explanation, which you should read carefully. <criteria list> (criteria list)
</criteria_liss list>
<Instruction
Your Task
Strictly follow each criterion in <criteria list> to evaluate how <target <target_article> article> meets that criterion. You must:
1. Analyze Each Criterion:
For each item in the list, think about how the article meets the requirements of that criterion.
2. Analytical Evaluation: Combine the article content, the task, and the user persona to analyze the article's performance for that criterion, pointing out both strengths and weaknesses.
3. Scoring: Based on your analysis, give score between 0 and 10 (integer) for the article's performance on that criterion.
Scoring Rules
For each criterion, give a score between O and 10 (integer), The score should reflect the quality of the article's performance:
0-2 points:
Very poor. Almost completely fails to meet the requirement.
2-4 points: ings.
Poor. Meets the requirement only partially, with significant shortcom-
4-6 points: Average. Basically meets the requirement; neither particularly good nor bad.
6-8 points:
Good. Mostly meets the requirement, with notable strengths.
10 points:
Excellent/Outstadding. Fully exceptionally meets the requirement.
Output Format Requirements
Strictly follow the <output format> rion. Do not include any irrelevant content, introductions, conclusions. Start from the first dimension and output all dimensions and their criteria in sequence: </Instruction>
<output_format>>>> below to output the evaluation results for each crite-
<output_formaz
"goal alignment";
"criterion": "[The text of the first Goal Alignment criterion]",
"analysis"; "[Analysis]",
"target_score". "[integer score 0-10]"
"criterion": "[The text of the second Goal Alignment criterion]",
"analysis"; "[Analysisl", "target_acore": "[integer score 0-10]" b
1,
"coneent alignment";
"criterion": "[The text of the first Content Alignment criterion]",
26

---

Published as a conference paper at ICLR 2026
"analysis"; "[Analysisl", "target_acore"" "[integer score 0-10]" b
1, "presentation__it"" =it":
"criterion": "[The text of the first Presentation Fit criterion]",
"analysis"; "[Analysisl", "target_score": "[integer score 0-10)" },
},
"act iettioiabiityy htt.tttty.atiiatttyy...
"criterion"; "/The text of the first Acticnability Practicality criterion]", "analysis"; "[Analysis]", "target_score"; "[integer score 0-10]"
b
<oomtput_format>
J.2
PROMPTS IN QUALITY EVALUATION
Prompt for Quality Dimension Weights Allocation
You are an experienced expert in evaluating research reports. You excel at deeply under- standing the goals, challenges, and core value points of given research task, and setting dynamic, reasonable, and well-justified dimension weights for subsequent report quality evaluations.
</system_role role>
<user_ spee__promp>>>
Here is a deep research task as follows:
<task>
"[ (ttask_prompt)]
</task>
<instruction>
Background: The research team will conduct an in-depth and comprehensive investigation
based on the above <task> and eventually produce high-quality research report.
Your Task: As an evaluation expert, you need to set the evaluation dimension weights specifically for this <task>, The evaluation will be carried out around the following three
dimensions:
Whether the report provides sufficient depth and unique insights. Logical Coherence: Whether the report's 's reasoning framework is rigorous and its logical derivation coherent.
1. Depth & Insight:
3. Clarity & Readability: Whether the report's language, information presentation, and formatting clear and easy understand, allowing readers absorb the content smoothly.
27

---

Published as a conference paper at ICLR 2026
Evaluation Formula: Total Score (Depth & Insight weight;) (Logical Coherence weightz) + (Clarity & Readability weightz). (Note: The sum of all weights must equal exactly 1.0)
Core Requirements:
1. Analyze the Task in Depth: Carefully study the <task> content, implicit objectives, potential challenges, and the core value of the deliverable.
Dynamically Allocate Weights: Based your analysis, assign weights the three dimensions (use decimals between and I, e.g., 0.4). The key is to understand that different tasks emphasize different aspects weights must be adjusted flexibly based on task characteristics, rather than being fixed.
Explain the Allocation Rationale: Your analysis (<analysis>) must clearly and specifi- cally explain why each dimension is given its corresponding weight and directly link your reasoning to the requirements and characteristics <task>, This the key criterion for evaluating your work quality.
Standardized Output Format: Strictly follow the example format below - first output the detailed rationale in <analysis>, then provide the weight allocation result in <json output>.
</instruction>
<examples_ eexamples_rationale>
Below are two examples, which demonstrate how to adjust dimension weights according the nature of the task and explain the reasoning. Please focus on learning the thinking process and analytical approach shown in the examples, rather than simply copying their content or numerical values. </examples rationale> _
Please strictly follow the above instructions and methodology. Now, for the following spe- cific task, begin your work:
<task>
</task>
Please output your <analysis> and <json_ous output>> </user_ pnser_prompt>
Prompt for Depth & Insight Criteria Generation
You are an experienced expert in evaluating research reports. You excel at breaking down abstract evaluation dimensions (such as "Depth & Insight") into actionable, task-specific, and clear criteria, assigning reasonable weights and explanations for each. </system_roles
<user Dnser_prompt>
Background: We are evaluating a research report based on three dimensions: Depth & Insight, Logical Coherence, and Clarity Readability.
Depth Insight: Whether the report provides sufficient depth and unique insights.
2. Logical Coherence: Whether the report's reasoning framework is rigorous and its logical derivation coherent.
Clarity & Readability: Whether the report's language, information presentation, and formatting are clear and easy to understand.
<task>
<task>>
28

---

Published as a conference paper at ICLR 2026
<instruction
Your Goal: For the Depth & Insight dimension of this report, develop a detailed, specific, and highly task-targeted set of evaluation criteria. You need to:
1. Analyze the Task: Examine <task> in depth and identify where deep analysis, logical reasoning, insight extraction, or value judgment are required to demonstrate insight.
2. Formulate Criteria:
Based the analysis, propose concrete evaluation criteria focusing on analytical depth, logical rigor, originality, and value of conclusions.
3. Explain Each Criterion: Provide a brief explanation (explanation ion) for why this criterion is important for evaluating Depth & Insight for <task>.
4. Assign Weights: Assign a reasonable weight (weight) to each criterion, ensuring the weights sum exactly 1.0. The weights should reflect the relative importance of each criterion within the Depth & Insight dimension.
5. Avoid Overlap: Clearly focus only on criteria relevant to Depth & Insight, avoid- ing aspects of Logical Coherence (structure) or Clarity & Readability (language, formatting).
Core Requirements:
Stay Task-Specifi::
The analysis, criteria, explanations, and weights must directly relate to the task's core requirements and characteristics.
2. Go Beyond the Surface: The criteria should assess analytical depth, reasoning rigor, originality of insights, and value of conclusions not just listing informa- tion.
3. Provide Strong Rationale: The <analysis> section must clearly explain the overall approach designing the criteria and weights, linking <task>, Each explanation must justify the criterion.
4. Ensure Reasonable Weighting:
Weight distribution must be logical, reflecting the relative importance of each criterion in showing insight.
5. Standardized Output Format: Strietly follow the format below: output <analy <analysis> first, then <json_output>,
</instruction>
<example_rationall rational>
Below is an example demonstrating how to design Depth & Insight criteria. Focus on the thinking logic and analytical approach rather than copying its contents or weight numbers. </example <eexample_atioonal>>>>> rational>
Please strictly follow the above instructions and methodology. Now, for the following spe- cific task, begin your work: <task>
[[task_promp)
</task>
Please output your <analysi and <json_outputs
Scoring Prompt for Quality
<system role>
You are strict, meticulous, and objective expert in evaluating the quality of research articles. You excel at deeply evaluating research articles based on specific quality assessment criteria, providing precise scores and clear justifications. </ </system_roles
<user_prompt>
29

---

Published as a conference paper at ICLR 2026
Task Background
You are given an in-depth research task. Your job is to evaluate research article written for this task. We will evaluate across the following three dimensions: Depth & Insight, Logical Coherence and Clarity & Readability. The task is as follows:
<task>
"(task_pa prompt)"
</task>
Article to be Evaluated <target_article "(article)" </target_article>
Evaluation Criteria
You must evaluate the article's performance for each criterion the list below, outputting your analysis and then assigning score from 0-10. Each criterion includes its explanation, which you should read carefully.
<eriteria_list (criteria_list] sicriteria_list>
<Instruction>
Your Task
Strictly follow each criterion in <criteria_list>
to evaluate
how
<target_article> meets that criterion. You must:
1.
Analyze Each Criterion: For each item in the list, think about how the article meets the requirements of that criterion.
2. Analytical Evaluation: Combine the article content with the explanation of the criterion to analyze the article's performance for that criterion, pointing out both strengths and weaknesses.
3. Scoring: Based on your analysis, give a score between 0 and 10 (integer) for the article's performance on that criterion.
Scoring Rules
For each criterion, give score between and 10 (integer). The score should reflect the quality of the article's performance:
0-2 points:
Very poor. Almost completely fails to meet the requirement.
2-4 points: ings.
Poor. Meets the requirement only partially, with significant shortcom-
4-6 points: Average. Basically meets the requirement; neither particularly good nor bad.
6-8 points:
Good. Mostly meets the requirement, with notable strengths.
Excellen//Outstadding, Fully exceptionally meets the requirement. Output Format Requirements
8-10 points:
Strictly follow the coutput_foomaet below to output the evaluation results for each criterion. Do not include any irrelevant content, introductions, or conclusions. Start from "criterion I" and output all criteria in order: </Instruction>
<output_format
"depth "eptghtinsigh"""
"criterion": "[The text of the first Depth & Insight criterion]",
30

---

Published as a conference paper at ICLR 2026
"analysis"; "[Analysisl", "target_score"" "[integer score 0-10]"
"criterion"; " "[The text of the second Depth Insight criterion]",
"analysis": "[Analysis]",
Starget_score", "[integer score 0-10]"
1,
"logi cal c
[
"criterion": "[The text of the first llogical cal Ccherence criterion]",
"analysis": "[Analysisl",
"target_ score"; "[integer score 0-10]"
"criterion"; "[The text of the second Logical Coherence criterion]",
"analysis"; "[Analysis]", "target_score "[integer score 0-10]" b
b.
"clarrty_readailltyy
"criterion"; "[The text of the first Clarity Readability criterion]", "analysis"; " "[Analysisl", "target_score "[integer
0-10]"
"criterion"; " "[The text of the second Clarity & Readability criterion]", ""nalysis": "[Analysis]", "target_score": "[integer score 0-10]"
</output_forme format>>
J.3
PROMPTS IN RELIABILITY EVALUATION
Prompt for Claim Extraction
You will see research report, and your task is to extract only all verifiable factual statements (factual claims) from the text.
# Definition of Factual Statement factual statement is verifiable claim about the objective state of the external world. It describes facts that have already occurred, quantifiable data, recognized classifications, or scientific laws not the author's subjective opinions, inten- tions, plans, or predictions about the future, nor descriptions about the report's 's own plans or structure.
# Guidelines for Identifying Factual Statements
# Types to Extract (Examples)
Specific data and statistics: "In 2023, global electric vehicle sales reached million units."
31

---

Chinese society can be divided Published as a conference paper at ICLR 2026
Past historical events:
"The company founded in Shanghai, China, in 2010."
Recognized classifications or definitions: "Li Qiang constructed socioeconomic status index (SES) based on income, education, and occupation, dividing society into seven classes [15]."
Cited research findings: "Studies show that more than eight hours of sleep are critical for memory consolidation [8]."
# Types to Exclude (Examples)
Goals and intentions: Any statement describing the "purpose," "goal," or "aim" of this document or project.
Example: "The goal of this report is to systematize personal creative activi- ties." or "This project aims to verify the small-revenue model."
Plans and proposals: Plans about future actions, strategies, content.
Example: "The content pillars include: travel sketch diaries, process break- downs, tool reviews..." or "We will execute this plan in three phases."
Self-referential statements about the document: structure content.
Statements introducing the report's
Example: "This report is a three-month brand and operations execution man- ual for..." "Chapter 3 will discuss market analysis detail."
Predictions and speculations: the future.
Estimations guesses about what might happen in "This could create new business opportunities."
Example:
"This strategy expected increase user stickiness by 20%.." " or
Opinions and recommendations: The author's subjective judgments, opinions, or suggestions.
"We believe this is a key breakthrough." or "Therefore, we recom- mend adopting Plan A."
Example:
Research methods: Descriptions of how the research or work will be conducted.
Example: "This study will adopt mixed-methor approach combining quali- tative and quantitative analysis."
# Extraction Rules and Output Format For each factual statement you find, determine whether it includes reference citation, and extract it as a (fact, ref idx, url) triple. Citations in the text may appear in the following forms:
1. A piece of text space + number, for example: "Li Qiang constructed socioe- conomic status index (SES) based on income, education, and occupation, dividing society into seven classes 15"
2. A piece of text [number(s)), for example: "Li Qiang constructed socioeconomic status index (SES) based on income, education, and occupation, dividing society into seven classes [15]"
3. A piece of text + [number())tssome line numbers etc.)], for exam- ple: "Li Qiang constructed socioeconomic status index (SES) based on income, education, and occupation, dividing society into seven classes
4. [Cited (citaton link), for example: "According A Guide to Social Class in Modern China](tttps: / www..chinafi.i. saa--ssocallccaac www.--aaaaaseeccaiieiioooo into nine classes"
[Chi-
naFile:
When extracting, pay attention to the following:
1. The extracted fact should be complete, understandable statement - not just phrase fragment.
32

---

Published as a conference paper at ICLR 2026
2. If fact cites multiple references, output multiple triples. For example, if it cites two references, output (fact, ref idx 1, url_1) and (fact, see_idx__2__ 2, url_2),
3. For the third form of citation, only take the first numeric part as ref _dx, ignoring indicators of specific locations. For the fourth form (where the source and link appear directly in the text), set ref _d_ uniformly to 0.
4. If a factual statement has no citation, set both ref idx and url to empty strings ""
Output Requirements: You should return JSON list, where each item one triple. For content you are unsure about, err on the side of caution - it's better to miss something than mislabel it. If there are no factual statements in the article, return an empty list []. JSON Example:
"fctt": "Text from the original article, use full-width Chinese quotation marks, escape English quotes with a single backslash",
"ref_idx"; "The index the cited reference the reference 1ist for this statement; Teave empty none", "uri": "The link of the cited reference (extracted from the report's reference 1ist or from the inline citation), leave empty if none"
"fact"; "In 2023, global electric vehicle sales reached 14.1 million units.", 12, "url": "https:/ /tta:://iaa.aaee hhtp::///eecooorrprrrrr/aaaarrrr
"fact": "Tesla went public NASDAQ in 2010.", "ref_idx": "" "url"; ""
"fact": "Studies show that than eight hours of sleep significantly enhances memory consolidation.", "ref_idx": 5,
"url": "https:
http:://o.ooooooo1.00000 101945"
"fctt: "According to UNEP
(https: http://wwwwww.....eee.ee
gl cbal greenhouse gas emi mmissions reached record high in 2023.4 "ret idx": 0, "uri":
Below the main text the research report: (report_text Now start extracting, and directly output the JSON list do not output any small talk explanation.
33


Reviews:
[{"review_id": "7hhmtquv48", "reviewer": "Reviewer_maAF", "summary": "Introduces a new benchmark and evaluation framework to address a gap in assessing Personalization in Deep Research Agents.\nPersonalized Deep Research Bench is a benchmark specifically designed to evaluate personalization in DRAs. Contains 250 user research task pairs.  PQR Evaluation Framework focuses on assessing the personalization alignment of a DRA output given a persona. Incorporates Goal Alignment, Content Alignment, Presentation Fit, and Actionability & Practicality.", "questions": "Any details of treatment for bias and reproducibility when running experiments?\nHow is the \"user’s evolving interests, habits, and implicit preferences\" incorporated in the profile/evaluation?\nAny instances of conflicting signals in the user profiles? How does the DRA handle such cases?\nHave you studied the factors that influence the dynamic weight allocation when running eval framework?\nWhat is the cost to run the benchmark? How can we scale it?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea here is good. Objective measures of Research Agent outputs are obviously important, but there is value in customizing the research process and the resulting artifact to match the sensibilities of the user. So the work addresses this particular gap.\n\nI expect the benchmark itself would be of some value to the research community. In particular the inclusion of structured and context data in user profiles, and the diversity of tasks. \n\nThe personalization metric is well defined and the evaluations presented are comprehensive.", "weaknesses": "It would be better if the personas were developed from real world interactions, the authenticity is limited. The sample size of 25 profiles is also relatively small for a large-scale, generalizable benchmark.\n\nUse of LLMs for evaluations. I know its common practice now but you need to address for bias and reproducibility. \n\nWhile I appreciate the definition of the personalization metric, this is a complex topic and would help to look for validation in prior research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V7TRNO2uET", "reviewer": "Reviewer_8bjw", "summary": "The paper introduces a benchmark for personalized deep research agents with 25 real human participants created persona profiles. The authors designed a new evaluation framework for this benchmark that jointly measures personalization, content quality and factuality.", "questions": "1. Can you show the correlation of your benchmark with some personalization benchmark and research agent benchmark?\n\n2. How to ensure that llm as judge are aligned with users behaviors?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Real human persona and questions and tasks are designed by human expert that increases the credibility of the benchmark in reflecting real user behaviors.\n\n2. Comprehensive analysis across different models and agents.", "weaknesses": "1. The motivation for combining personalization and deep research into one benchmark is lacking. These two aspects are largely orthogonal to me as they measure different things. The authors did not explain why a fused benchmark is necessary. One can imagine that combining the scores of existing personalization and deep-research benchmarks are already a good metric for measuring personalized deep research agent. There is no correlation analysis showing that performance on this benchmark differs from performance on either personalization benchmarks. For example, I want to see that a model performs well on both personalization and deep research agent benchmark, but performs poorly on your benchmark. Without such evidence, the benchmark’s purpose and contribution remain weak.\n\n2. LLM as judge bias. For personalization benchmark, it's important to show that llm as judge are unbiased, potentially by comparing with the real participants' and users' feedback on some models, so you can show that llm evaluation are aligned with actual user behaviors. The section 5.5 has a alignment with some human rating, this does not address the problem, since it's a personalization benchmark rather than a rlhf benchmark. It would be great to see the actual alignment with the actual user who generated the persona.\n\n3. Scale is somewhat limited. With only 25 profiles, it's hard to generalize to the whole populations and demographics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lDinRk5bzI", "reviewer": "Reviewer_vdTb", "summary": "The paper proposes a new benchmark for personalized deep research. Deep research is a popular mode of AI agents, where an agent utilizes several tools like iterative search, to answer complex question. Personalization in the context of deep research is underexplored. This paper presents a new benchmark to evaluate the personalization capabilities of LLMs in the context of deep research. Specifically, the paper focuses on evaluating 3 key aspects of deep research responses -- personalization, response quality, and reliability. The paper presents automated metrics to evaluate each of these dimensions.", "questions": "Please respond to the questions in the above sections.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper focuses on a relatively underexplored aspect of LLM evaluation -- personalization in deep research. The paper discusses the key drawbacks of existing benchmarks and builds on those aspects. \n2. The paper collects personalization data in a thorough manner by including people of diverse backgrounds and collect data from these folks over prolonged interactions.", "weaknesses": "1. I'm concerned about the privacy implications of the benchmark release. According to Table 5 in Appendix E, the persona collection process includes identity characteristics like name, age, gender, etc. These information along with others could often be enough to reveal the identity of an individual. \n\n2. The proposed evaluation process seems to be weak. Along the personalization dimension in Section 4.1, the authors ask the LLM to generate both the criteria and the weight of each criteria. Were these information reviewed by experts? Also, it seems unclear which LLM was used for this specific task. \n\n3. The paper misses out on key details in some sections like 4.2. Why did the paper choose not to use well-established factuality metrics like FActScore [1]? How are the verifiable claims extracted from the answers?\n\n4. How is $\\lambda_a$ in Equation 6 chosen?\n\n5. Most of the results presented in the paper are scores generated by LLM. In Section 5.5, the paper evaluates the correctness of these scores by comparing it human annotators. However, this task was simplified to a great degree where the annotator had to choose between a couple of responses. The paper compares LLM score agreement in this setting. This is not an accurate measure of correlation of the actual score (which are between 1-10). Even in this simplified setting, the correlation seems to be poor raising questions about the quality of LLM ratings. \n\n6. In Table 1, 3, and left figure of Figure 3, we observe that incorporating additional personal context doesn't change the performance too much, even the P-Score. Does this mean current LLMs aren't able to personalize based on the provided context? If not, how can the P-score be so high without any context?\n\n\n[1] FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Conditions for acceptance have been satisfied.
Decision: Accept (Poster)