VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications
Authors: Wei He
Yueqing Sun
Hongyan Hao
Xueyuan Hao
Zhikang Xia
Qi GU
Hui Su
Xunliang Cai
Venue: ICLR 2026 Poster (2026)
Keywords: llm agent
tool use
multi-turn interaction
real-world application
Primary area: datasets and benchmarks

Abstract: As LLMs with agentic abilities are increasingly deployed in real-life scenarios, existing benchmarks fail to capture their inherent complexity of handling extensive information, leveraging diverse resources, and managing dynamic user interactions. To address this gap, we introduce VitaBench, a challenging benchmark that evaluates agents on versatile interactive tasks grounded in real-world settings. Drawing from daily applications in food delivery, in-store consumption, and online travel services, VitaBench presents agents with the most complex life-serving simulation environment to date, comprising 66 tools. Through a framework that eliminates domain-specific policies, we enable flexible composition of these scenarios and tools, yielding 100 cross-scenario tasks (main results) and 300 single-scenario tasks. Each task is derived from multiple real user requests and requires agents to reason across temporal and spatial dimensions, utilize complex tool sets, proactively clarify ambiguous instructions, and track shifting user intent throughout multi-turn conversations. Moreover, we propose a rubric-based sliding window evaluator, enabling robust assessment of diverse solution pathways in complex environments and stochastic interactions. Our comprehensive evaluation reveals that even the most advanced models achieve only 30% success rate on cross-scenario tasks, and less than 50% success rate on others. Overall, we believe VitaBench will serve as a valuable resource for advancing the development of AI agents in practical real-world applications.

Work in progress.
VITABENCH: BENCHMARKING LLM AGENTS WITH VERSATILE INTERACTIVE TASKS IN REAL-WORLD APPLICATIONS
Wei H,2.2*t, Yueqing Sun" Hongyan Hao Xueyuan Hao"* Zhikang Xia", Qi Gu²t, Hui Su², Xunliang Cai2
Fudan University, "Meitaun LongCat Team
ABSTRACT
As LLMs with agentic abilities are increasingly deployed in real-life scenarios, existing benchmarks fail to capture their inherent complexity of handling exten- sive information, leveraging diverse resources, and managing dynamic user in- teractions. To address this gap, we introduce VitaBench, a challenging bench- mark that evaluates agents yersatile interactive tasks grounded in real-world settings. Drawing from daily applications in food delivery, in-store consumption, and online travel services, VitaBench presents agents with the most complex life- serving simulation environment to date, comprising 66 tools. Through frame- work that eliminates domain-specifee policies, enable flexible composition of these scenarios and tools, yielding 100 cross-scenario tasks (main results) and 300 Linglesceearro tasks, Each task derived from multiple real user requests and re- quires agents to reason across temporal and spatial dimensions, utilize complex tool sets, proactively clarify ambiguous instructions, and track shifting user intent throughout multi-turn conversations. Moreover, we propose aabric-based slid- ing window evaluator, enabling robust assessment diverse solution pathways in complex environments and stochastic interactions. Our comprehensive evalua- tion reveals that even the most advanced models achieve only 30% success rate on scesa-scenario tasks, and less than 50% success others. Overall, we believe VitaBench will serve as a valuable resource for advancing the development of Al agents in practical real-world applications
52 Non-trirking
Z
M
AI
42
Therking Mode:
scemura Ancrage
Z M
Cisco cowsers desin Recult)
Deettani-7 Apo
ons didioaa
tleut ail
4lmel 20O7
as Claddn
Figure 1: Overall performances on VitaBench, sorted by main results.
INTRODUCTION
Recent advances in large language models (LLMs) have significantly enhanced their complex rea-
soning and tool-use capabilities (Bai et al. 2025 Zeng et al. 2025: Li et al. 2025 leading to
'Equal contribution.
1 'Corresponding authors. Correspondence to: whe23 @ m. www.uaa.cddu.c..mmmmmmmmm muniO3@mmituancoom
'The code, dataset, and leaderboard available at wthhtps:://// wwwpccc..abbe.....com gith.b. io

---

Work in progress.
Environment
Trajectory
Rubric Checklist
User
Real-world
want pizza...
Ordered pizza.
Choose target...
Payment processed.
Application
Agent
R
Delivery In-store
OTA
Which restaurant...
W
x
x
Tool Graph
Database
W
4
x
Service Providers
The regular one.
W
X
Transactions
Read
Products
Let me check...
Other info.
Write
W.
Figure 2:
VitaBench sources tasks from real-world environments by composing interconnected tools, diverse user requests, and structured databases. Agents interact with users through multi- turn dialogue, while a rubric-based sliding- winddow evaluator tracks progress across the trajectory.
increased deployment of LLM agents in real-world applications. These improvements have simul- taneously driven the evolution of agent-centric benchmarks (Yao et al. 22224] Barres et al. [025 Lu
et al. 2025), progressing from simple task execution to complex multi- multi-turm//i interaction scenarios.
However, there remains a significant gap between controlled laboratory settings and real-world de- ployments that present inherently complex challenges. Early benchmarks (Qin et al. 2024 Patil
et al.; (025)) focused primarily on function-calling and parameter accuracy, introducing difficltty through increased tool counts distractors, yet overlooking the intricate interdependencies between tools and their environments. Some recent efforts (Yao et al., 2024 Barres et al. (025) have begun exploring real-world challenges, but often impose rigid domain-speciffe policies and constrained action spaces, overemphasizing cestructionfollowwiin many inadequately consider users as environmental components who bring inherent uncertainty, despite this being critical challenge for practical agent applications (Qian et al. 2025).
over autonomous exploration. Furthermore,
This motivates our central research question:
"What constitutes task complexity for agents in real-world applications?"
Drawing inspiration from task complexity theories that examine structural, resource, and interac- tion dimensions (Liu & Li, 2012). we identify three fundamental aspects that shape agentic task complexity: (1) reasoning complexity, measured by the volume of environmental information that agents must process and integrate: (2) tool complexity, quantified through modeling tool sets as graphs based on inter-tool dependencies, where the node cardinality and edge density reflect the structural intricacy to navigate; (3) interaction complexity, characterized by the challenges arising from diverse user behavioral attributes and conversational patterns throughout multi-turn interaction.
Building on this framework, we present VitaBench (short for Versatile Interactive Tasks Bench- mark) to measure agent's ability to handle the inherent complexity of real-world applications (overview in Figure D. We construct 66 tools across three domains-deliveryy in-store consump- tion, and online travel services and model their intrinsic dependencies policy information is inherently encoded. This allows agents to reason and explore autonomously without relying on domain-specific policies like 7-ench Yao al. 2024). This design also enables flexible composition of scenarios and toolsets, facilitating the creation of 400 evaluation tasks span- ning both single-scenario and cross-scenario settings. We derive each task from multiple authen- tic user requests and equip it with an independent environment containing annotated user profiles, spatiotemporal contexts, and comprehensive service databases. Given the extensive solution space of these instructions and environments where numerous valid pathways may exist, we introduce a rubric-based sliding window evaluator to assess the resulting long-horizon trajectories.
graph structure where
We evaluate multiple advanced LLMs on VitaBench, revealing that even the best best-performing model achieves only 48.3% success rate across our 300 single-scenario tasks, with performance plum- meting to 30.0% in cross-scenario settings where agents must navigate between different domain contexts and choose right tools from expanded action spaces (Figure D.. Our comprehensive anal- ysis validates the three- dineeddimnnional complexity framework, showing strong correlations between complexity metrics and task difficulty across domains. Through systematic failure pattern analysis, identify that reasoning errors dominate (61.8%) followed by tool usage errors (21.1%) and in-
The name "ita" derives from the Latin word for "I "Life", reflecting our facus life-serving applications.
2

---

Work in progress.
teraction management failures (7.9%), with agents exhibiting poor self-awareness and limited error recovery capabilities. Rigorous validation confirms the reliability of our evaluation components, establishing VitaBench challenging and reliable benchmark for advancing real-world agent pabilities. All code and data will be released to ensure reproducibility.
RELATED WORK
Table 1: Comparison of existing user interaction benchmarks across three complexity dimensions: reasoning, tool, and interaction. "V" indicates fully addressed, "X" indicates partially addressed, and "X" indicates not addressed. Detailed explanations for each trait are provided in Appendix
Reasoning Complexity
Tool Complexity
Interaction Complexity
Benchmarkk
Multifaccted Composite
Goal
Cross Depeadency Socnarios
Inter-tool
# Turns (approx.)
User
Behavior
Information
Objective
Ambiguity
Tools
Profile Attributes
TooTTalk (Farn & Shin
2023
28
X
2, 10
IN3 (Qian
2.10
Wang e au 2024b
x
X
[2. 10
X
X
ToolSandbox
/
10,
DialogTool WWag el m
2025
31
V
/
10,30
J
UserBench (Qan el al
X
X
10. 30
Bench Yáo cl 2024
X
30, 50
7 BeBcch (Hrres of al: 2025
38
/
X
30, 80
VitaBennh (ours)
66
[00, 100)
Early tool-use benchmarks (Huang et al., 2024) Qin al. [024] Patil et al. 2025) primarily focused on single-turn API calling accuracy, overlooking the inter-tool dependencies and dynamic interac- tions with users that characterize real-world applications. While recent work has recognized the need for evaluating advanced reasoning, tool manipulation, and interaction abilities, current bench- marks typically address these dimensions in isolation rather than comprehensively. Table Vable|eooppares compares prominent agent-usce interaction benchmarks across proposed task complexity framework.
ToolTalk (Farn Shin 2023) first introduces multi-step tool execution through conversational inter- faces but relies predefined dialogue trajectories, limiting agent autonomy. While MINT (Wang) et al. 2024b) emphasizes natural language feedback to guide agents and IN3 (Qian et al. 2024) fo- cuses detecting implicit intentions, both of them operate in relatively constrained agentic settings. More comprehensive frameworks like ToolSandbox (Lu et al. 2025) and the 7-bench family (Yao)
al. 2024 Barres et al. 2025) pioneer stateful execution and model tool interdependencies, yet
constrain agents through verbose policies rather than allowing truly autonomous exploration. Di- alogTool (Wang et al. 2025) explores role-playing for engaging users but focuses primarily on agent-side capabilities, while UserBench (Qian et al. 2022) uniquely captures preference-driven in- teractions, though with limited task complexity otherwise. Several works (Yang et al., 2024 Wang et al. (2024a) also investigate agents' abilities to recognize incomplete conditions and proactively seek missing information. However, none of these benchmarks simultaneously challenge agents across multiple complexity dimensions. Our work aims to bridge this gap with VitaBench, which presents information enviromments requiring agents to autonomously explore, dynamically in- teract with diverse users, and navigate intricate tool dependencies to address real-world demands.
3
VITABENCH: A BENCHMARK FOR VERSATILE INTERACTIVE TASKS
3.1 FORMULATION
The POMDP Formalism. We formalize the set of distinct environments as For specific environment E E, model the agent task partially observable Markov decision process (POMDP) (U, S, A, O,,,), Tr) with instruction space U. state space S, action space A, observation space O, state transition function T S A S, and reward function S A R.
The agent interacts with both databases (through API tools) and a simulated user. Accordingly. the action space A consists of two types of actions: tool invocation and interactive dialogue with the user. The state space S comprises the state of the database and the user state, i.c., S = Sab @ Sper- The observation space O includes the database feedback after tool calls and the conversation history
3

---

Work in progress.
with the user, i.e. O Oab & Ouser- The state transition function T decomposes accordingly: API calls follow deterministic transitions Tap implemented as Python functions, while user interactions follow stochastic transitions Tuser implemented using language model.
Given an instruction u E 2. the initial state so represents the token sequences of the given prompt and the initial state of the database. The agent receives initial observation On O, which typically includes the first-round user request and the available tool sets. The LLM agent, parameterized by 0. generates an action d1 (To-)on) based on its policy иT.. Subsequently, the state transitions to 81 E S, and the agent receives feedback 0D E O. At each step I, the agent acts based on the current observable history, which can be denoted as (op, 01, O1..... Qt-1. 0t-1), generating action a Te Te(-|o0. Q1, 01..... Qt-1, 0t-1). The agent continues interacting with the environment until the task completed or the maximum number of steps is reached. From the environment's perspective, the complete state transition trajectory can be represented as:
Su 01 $1, 022 $2,.... ar, 8T) (T(TTe.u).
(1)
where T denotes the total number of interaction rounds. Note that the trajectory captures the complete state transitions, while the agent only has access to partial observations Or derived from states s. The reward r(e, u, 7) E [0, 1] computed after the interaction ends.
Agentic Task Complexity Framework. Building upon the POMDP formalism and drawing in- spiration from multi- pesti-perspecctiee complexity frameworks (Liu & Li 2012), we formalize task com- plexity along three dimensions that capture the challenges agents face in real-world applications:
Cassk (Creason Gooo, Cinternet.
(2)
Reasoning complexity Creason quantifies the cognitive demands of processing extensive environ- mental information under partial observability. We characterize this through the entropy of the observation space (O)) and the degree of partial observability " 1 TST ues indicate greater uncertainty in state estimation. Building on this framework, we construct large-scale databases and composite tasks with multiple explicit and implicit reasoning points.
O where higher val-
Tool complexity Gool captures the structural intricacy of navigating interconnected action spaces. We model the toolset as directed graph G E)) where vertices represent individual tools and edges encode inter-tool dependencies. Complexity emerges from graph cardinality [V), edge IIT the coverage ratio VV
density p = |
|Vual of task-relevant subgraph. Cross-scenario settings
further amplify this by expanding the action space A across multiple domains.
Interaction complexity Cinternct reflects the challenges of managing dynamic multi-turn conversa- tions with users. User profiles encode personal attributes (e.g., gender, age, dietary restrictions) that influence task requirements. Behavior attributes introduce variability in cooperation levels and goal ambiguity, necessitating proactive clarification. Moreover, real-world users exhibit dy- namic states Swee that evolve throughout the interaction, requiring continuous strategy adaptation.
3.2 BENCHMARK CONSTRUCTION
We construct VitaBench through systematic pipeline illustrated in Figure Specifically, this process can be divided into two stages:
Stage I: Framework Design. We construct VitaBench through systematic abstraction of real- world life-serving scenarios across three domains: Delivery (food and product delivery), In-store Consumption (dining and other services), and Online Travel Agency (OTA) (hotel bookings, at- traction reservations, flight and train ticket management). By referencing existing application im- plementations, we derive simplified API tools that capture essential functionalities. We model inter-tool dependencies as directed graph G = (V. E) and augment tool descriptions with pre- conditions (states required before execution) and post-conditions (expected outcomes after execu- tion). This graph-based design naturally encodes domain rules into tool structures, eliminating the need for verbose policy documents while simultaneously increasing reasoning complexity and fa- cilitating cross-domain composition. For instance, modify requires prior execution of get order detail to obtain necessary information, reflecting natural workflow dependencies. The complete toolset used in VitaBench is documented in Appendix
4

---

Work in progress.
H
Tool Definition
Dutabase
Versatile Interactive Tasks
Name: modify order
hotel_1d: 6881
Env. Information
Instruction
trencondition:
valid order id valid nodify request
address: Shanghas,
Hotels (target/distr.)
The requests book their usual hotel with river view and romantic dianer for him and his wife with budget of $200 near the Huangpu River where they first
Post-condition:
type:
Restaurants (large//distr.)
Rooms (target/distr))
updated (unpa(a)
price: $280-$580
payment chack
PPacaaess (larget/distr.)
Parinetors:
tag: river view,
Transacttoo history
orden_id
(Live APIs)
Nx (Target) &e MX x (Distractor)
System time (June 24)
10th anniversary celebration.
Weather (Sunny, clear)
Tool Dependency Graph
8 User Profile
Rubrics
hotel_Search
user_ history
(coos--eeeario)
Age: 35-45
/ Hotel (River-view suite, one, 27 check-inn)
Gender: Male
lotel_infs
addr2coard
got_ nearby
Job: Professor
/ Romantic Dinner Booked
($200 budget, couple package, June 7PM, non-spicy) / Weather-dspendout
Dietary Restriction:
create_onde order
get
store info
Prefer non-spicy food
Persona: ooriente
(rooftop seats forecast
pay_ore order
get date
book seat
Patience: High
clear; otherwise indoor)
Figure 3: Overview of the VitaBench construction pipeline and simplified cross-scenario example.
To capture the inherent uncertainty in real-world interactions, we implement a user simulator follow- ing Yao et al. (2024). The simulator receives complete instructions containing multiple requirements but reveals them progressively to agents, and provides implicit constraints only upon inquiry. We configure each simulated user with unique profiles and behavioral attributes, employing prompt- based constraints to maintain persona consistency while minimizing critical errors that would im- pede task completion (validated in Section 5.1). Since fully unconstrained user behavior would introduce excessive randomness, our simulator adopts controlled design to provide a balance be- tween realism and evaluative stability, enabling fair and reproducible comparisons across models. Note that while user profiles are accessible to agents, we establish knowledge boundaries to reflect realistic scenarios-for example, agents cannot directly access dietary restrictions but must infer them from order history or user responses.
Stage II: Task Creation. Our data collection pipeline consists of four components: user pro- files, task instructions, environmental information, and rubrics. User profiles derive from au- thentic platform data, which we anonymize and enrich to create distinct personas with varied personal attributes and communication styles. These attributes encompass emotional expressions (e.g., impatient, anxious, indifferent) and interaction patterns (e.g., detail-oriented, dependent, logical), leading to diverse conversational dynamics throughout multi-turn dialogues. Task in- structions synthesize multiple real user requests into composite objectives, which we manually review and refine to ensure clarity and feasibility. Instructions either coordinate multiple sub- goals within a single domain or span across different domains in cross-scenario settings, requir- ing agents to navigate between distinct contexts. For environmental data, combine service provider and product information from real-world life-serving platforms with model-generated syn- thetic augmentation under human supervision. We deliberately intermix target options that satisfy all constraints with distractor options that violate specific requirements, creating extensive search spaces with numerous candidates while maintaining only a handful of valid solutions per task. Additionally, we generate transaction histories to support requirements involving consumption pat- terns (e.g.. "order the same meal as last time" or "book my usual hotel"). We iteratively refine each task through multiple trials with human verification, eliminating ambiguities while preserving multiple valid solution pathways. Through this process, we construct 400 tasks with comprehensive databases detailed in Table 2 where individual tasks typically involve 5-20 service providers and can include over 100 products in certain cases.
Table 2: Data statistics of VitaBench.
Cross-Scen. Delivery In-store OTA
Databases
Service Providers
1, 123
410
611
137
Products
6, 3,946
788
3,277
9,693
Transactions
417
151
API Tools
Write
27
Read
33
General
6
6
Tasks
100
100
100
100
5

---

Work in progress.
3.3 RUBRIC- SLIDING WINDOW EVALUATOR
Evaluating long-form agent trajectories presents unique challenges due to their extensive length and multiple valid solution paths. While Yao et al. (2024) rely on predefined database state comparisons, such methods cannot capture nuanced requirements such recommendations or planning behav- iors that leave final states unchanged, nor provide supervision for intermediate transitions. Recent rubric-based evaluation methods (Arora et al. 2025 Ruan al. 2025) inspire approach by de- composing complex goals into atomic criteria, enabling comprehensive requirement coverage. With durefully-designed rubrics, LLM-as-a-Judge can effectively replace fine-grained human judgments while maintaining high accuracy. To address the challenge that multi-turn trajectories often exceed context lengths, we propose sliding window evaluator that processes trajectories in sequential seg- ments while maintaining continuity through persistent rubric state tracking.
We manually design rubrics R {ri...., r for each task, comprising atomic criteria derived from task information (e.g.. "restaurant within 500m", "user only eats vegetarian food"). Each trajectory is divided into overlapping windows W, of w consecutive turns, with adjacent windows sharing 8 turns to ensure information coherence. When processing each window, the evaluator ex- tracts rubric-relevant information and propagates forward to enable consistent cross-window judg- ments. The evaluator maintains a state vector E 0, 1st that persistently records criterion satisfaction across windows once a rubric item r, is satisfied in any window, 8y is marked as 1, and if a previ- ously satisfied criterion is later negated, the corresponding state will be reset to 0 accordingly. For benchmark evaluation, we adopt a strict all-or-nothing scoring where success requires satisfying all rubric items: score = 1 j $j k:). Nevertheless, the fine-grained rubries enable detailed scoring analysis for identifying trajectory differences, providing valuable dense signals for reinforcement learning. Human evaluation yields strong inter-rater agreement with Cohen's 's K 2 0.81 (Cohen 1960) as shown in Section 5.1 validating the reliability of our approach.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
Models. We evaluate various state-of-the-art proprietary and open language models for agents. The complete list of evaluated models is provided in Appendix [ The leaderboard is divided into thinking and non-thinking model categories. For hybrid models that support toggling between two modes, we evaluate the think-on and think-off configuration in two categories.
Methods. The language agents implemented as function-calling agents, with all tools provided in the OpenAI tool schemas. Note that we use the minimal agent design because our focus is to evaluate the intrinsic capabilities of foundation models without introducing confounding factors from rapidly evolving agentic systems. We do not limit the number of interaction rounds for agent models, and the task terminates when the agent outputs "4 SSOP # or encounters failure. The user simulator is implemented using gpt-4. using c claude-33 7-sonnet to avoid overlap with the evaluated agent models. For main results, each task is run four times with consistent LLM temperature of 0.0 to promote deterministic outputs. The prompt templates we used for agent, user and evaluator are detailed in Appendix E An analyse of representative cost and latency statistics is provided in Appendix D
000005504-20 The evaluator is implemented
Metrics. For the results from four runs, we report Avg@ 4, Pass@44 and Pass"4 metrics averaged across tasks. Pass @ & represents the probability that at least one out of A i.i.d. task trials is successful. Pass"k k represents the probability that all k i.i.d. task trials are successful (Yao et al. 2024).
4.2
MAIN RESULTS
Table 3
presents comprehensive evaluation results on VitaBench. We can observe that:
Real-world tasks pose great challenges for current agents. Performance varies significantly across domains and correlates strongly with environmental complexity. Cross-scenario tasks expose

---

Work in progress.
Table 3: Performance comparison of non-thinking and thinking models across different domains.
Cross-Scenarios
Delivery
In-store
OTA
Models
Avg @1
Pass @41
Pass
Avg
Pass @4
Pass
Avg @4
Pass @41
Pass "1
Avg @-1
Pass @41
Pass "4
@1
"4
Non-thinking Models
DeepSeek-V3-tt V3-0324
3.8
12.0
0.0
25.3
53.0
5.0
34.3
71.0
5.0
10.3
26.0
1.0
Qwen3-32B (w/o thinking)
4.0
12.0
0.0
16.5
37.0
3.0
21.3
47.0
2.0
3.0
11.0
0.0
GPT-5 (minimal)
4.0
9.0
0.0
30.0
64.0
6.0
27.0
60.0
2.0
7.8
22.0
0.0
Gemini-2.5-Fin (think off)
5.8
17.0
1.0
31.0
65.0
6.0
22.8
46.0
3.0
18.5
44.0
1.0
Doubao-Seed-1.6
10.5
29.0
0.0
37.8
65.0
12.0
39.5
73.0
9.0
18.8
39.0
3.0
GPT-4.1
13.8
35.0
37.8
67.0
11.0
71.0
17.0
42.0
1.0
Qwen3-235B-A22B-- B.SSB-A22B-Insteeee 1.222--Inttruct2500
14.3
38.0
34.3
66.0
44.8
87.0
13.0
20.0
45.0
1.0
Kimi-K2-0905
15.5
39.0
2.0
35.3
68.0
9.0
42.5
78.0
10.0
22.0
46.0
4.0
DeepSeek-V3.. (w/o thinking)
16.3
40.0
1.0
34.0
67.0
6.0
42.5
76.0
7.0
18.3
47.0
1.0
GLM-4.5 (w/o thinking)
20.0
47.0
1.0
45,8
72.0
20.0
48.3
82.0
13.0
20.3
45.0
2.0
LongCat-Flash-hat
20.3
45.0
2.0
39.5
71.0
15.0
50.5
84.0
15.0
22.8
49.0
2.0
Claude 4- Sonnet (w/o thinking)
4-anne-
21.3
49.0
4.0
39.0
69.0
17.0
46.3
78.0
10.0
25.0
49.0
7.0
Claude 4.1-pus (wo thinking)
21.8
47.0
3.0
46.0
78.0
13.0
53.8
85.0
21.0
30.8
60.0
9.0
Thinking Models
Qwen3-32B (w/ thinking)
5.0
24.0
0.0
22.8
53.0
4.0
26.5
60.0
3.0
7.3
18.0
1.0
2e5 S-Flash (think on)
5.3
32.0
62.0
23.0
57.0
3.0
39.0
1.0
DeepSeek-RI 1-0528
14.5
39.0
40.3
72.0
11.0
41.3
79.0
7.0
13.0
32.0
2.0
Doubao-Secd Seed--eed--- 1-ed--.6-Thhae T-innining
17.0
42.0
1.0
30.3
59.0
10.0
43.3
78.0
10.0
18.0
45.0
2.0
Qwen3-235B-A228 1---5---2B----- 2
18.8
45.0
2.0
44.0
78.0
9.0
46.0
80.0
9.0
17.5
41.0
2.0
04-mini (high)
19.5
49.0
1.0
44.5
80.0
15.0
46.5
81.0
15.0
23.5
50.0
5.0
GLM-4.5 (w/ thinking)
22.8
48.0
2.0
44.5
77.0
14.0
52.8
80.0
22.0
28.8
55.0
7.0
Claude-4-Sonaa (w/ thinking)
23.0
51.0
6.0
46.0
78.0
15.0
51.5
80.0
21.0
29.0
55.0
9.0
Gemini-2.5-Pro
23.5
53.0
5.0
49.0
81.0
16.0
43.8
78.0
12.0
26.5
54.0
6.0
LongCat-Flaad Thinkii
24.3
42.3
71.0
13.0
56.8
85.0
25.0
59.0
6.0
Claude-4.1 1.pus (w/ thinking)
29.0
47.5
80.0
78.0
20.0
57.0
9.0
o3 (high)
30.0
61.0
53.5
83.0
24.0
53.5
86.0
19.0
37.8
66.0
10.0
the most severe limitations: pared to over 50% in single-domain settings. This dramatic gap reveals fundamental deficiencies in navigating expanded action spaces and coordinating across distinet domains. Notably, task diffi- culty does not correlate with database scale the in-store domain, despite having far more products, proves easier than delivery settings. This counterintuitive finding shows how real-world complex- ity emerges: delivery tasks demand precise coordination of multiple items under strict constraints, while in-store operations remain straightforward despite larger candidate pools.
top-performing models achieve only 30.0% Avg score, com-
Exploration improves performance but reveals stability issues. The Pass@ k and Pass"k " metrics capture complementary aspects of model behavior. Pass@4 results show that increased sampling substantially improves completion rates, indicating that complex environments reward exploration, which suggests promising directions for RL approaches. However, Pass" 4 metrics reveal concerning instability, with even top models dropping near-zero consistency rates. To further validate this observation, we evaluate representative models with & = 32 samples (Figure 4 confirming that while exploration yields marginal gains, fundamental stability challenges persist even for leading agentic models like Claude-4-Sonnet.
Thinking mechanisms improve both effectiveness and efficiency. Thinking models generally outperform their non-thinking versions, with improvements such as Claude-4.1-Opus increasing from 21.8% 29.0% and GL.M-455 from 20.0% to 22.8% Moreover, thinking mechanisms lead to efficiency improvements, as shown in Figure 5 where thinking models tend to achieve better performance with fewer turns on average. For instance, the overall trend demonstrates that higher- performing models require fewer interaction turns, with thinking models achieving an average per- formance of 23.8% compared to 17.9% for non-thinking models, while maintaining comparable turn counts (61.1 vs 69.9 turns respectively). This efficiency gain stems from two factors: the bet- ter decomposition of complex multi-step plans and more targeted user interactions through precise clarifying questions.

---

Work in progress.
30.0
Thinking Non Thinking
Performance (Avg@
Guve-tt Open
50
Pass@k
25.0
Serth
Lengoe flad-Thheddrs
doSknngt ALMAN jort
22.5
dedi open (oft
Uluude4-5 Passggk
GUM-
GPT4 Pk
Claude-455 Pass"k
30
17.5
5
GPT4.1 Pass
15.0
BonsSeek
@wenb Km
2714
0-
2
50
55
60
70
75
80
85
90
Number of Attempts
Number of Turns
Figure 4: Pass@k Pass"k performance.
Figure 5; Model performance vs. Turns.
DISCUSSION
5.1 RELIABILITY ANALYSIS OF VITABENCH COMPONENTS
Given that our benchmark incorporates model- components for user simulation and trajectory evaluation, we conduct reliability analyses to validate their effectiveness and stability.
Reliability of user simulator. We evaluate our user simulator across two critical dimen- sions: information fidelity and persona consis- tency. For information fidelity, two annotators assess 100 conversations examining adherence to task instructions and user profiles, absence of hallucinations, and contextual relevance. As shown in Figure 6 a). the simulator achieves high fidelity with 9.48/10 10 average score across all scenarios. Minor deviations manifest as nat- ural conversational variations (c.g., "cannot eat spicy" vs. "prefer non-spicy food") that enhance di- alogue authenticity without compromising task requirements. Notably, the simulator appropriately responds "I don't 't know" when queried about unprovided information, maintaining strict source fi- delity. For persona consistency, we test five distinct personality types across 100 conversations, measuring behavioral alignment through language style, decision patterns, and emotional expres- sions. 6 demonstrates strong gersona-behavion alignment averaging 9.34/10. Cooperative personas exhibit the highest consistency, aligning with LLMs' inherent collaborative tendencies, while scattered personas show lower controllability.
(a) Info Fidelity
(b) Persona Consistency
9
9/5
26
+/1
0/5
5.9
33
Pranide Cop
Figure 6: User simulator reliability evaluation.
To further examine whether the user simulator implicitly favors agents with similar behaviors or reasoning patterns, we conduct a cross-model comparison by replacing GPT-4.1 with Claude-4- Sonnet the simulator in cross-scenario tasks.
Table Cross-model analysis of potential simulator-agent cooperation.
As shown in Table 4 GPT--4, performance re- mains nearly unchanged under different simulators, whereas Tluude--Sonnee exhibits a mild drop when interacting with simulator of the same family. indicates that any implicit cooperation between the simulator and the agent minimal.
User
Agent
Evaluator
Avg
Pass@4
Pass1]
GPT-4.1
GPT-4.1
Claude-3.7-S
13.8
35.0
GPT-4.1
Claude
Claude-3.7.S
21.3
49.0
4.0
Clande--8
GPT-41
Clande-3.7-8
13.8
34.0
Claude--SS
Claudo--S
Claude-3.7-S
19.5
45.0
Reliability of evaluator. We conduct ab- lation experiments to validate our rubric- based sliding window evaluator on GLM-4.5's cross-scenario trajectories. Table 5 compares four configurations against human-anmotated ground truth: (I) baseline with sliding window and rubric, (2) full trajectory with rubric, (3) sliding window without rubric, and (4) full tra- jectory without rubric. For configuration (3), employ external memory module main-
Table Ablation study of evaluator components. The "Score" refers to the evaluation score as- signed to corresponding GLM-4.5 trajectories af- ter applying the respective method.
Method
Score Task Acc.
Rubric Ace. Cohen's
Baseline
20.0
95.0
88.5
0.828
wio Sliding Window
19.0
90.0
87.6
0.604
wio Rubric Chocklist
91.0
22.0
0.018
w/o Both
82.0
32.0
0.067
8

---

Work in progress.
tain context awareness. The result shows that our proposed method achieves the highest agreement with human judgments (Cohen's 's k = 0.828), significantly outperforming methods without rubric structure (K 0.07). While full trajectory with rubric yields similar final scores (19% vs. 20%), the evaluation model's limited long-context capability hinders accurate assessment of all rubrics in the full trajectory. The sliding window design effectively handles this while maintaining 95% task-level accuracy, confirming the reliability of our approach.
Since the evaluator is also an LLM, natural concern is whether its reasoning patterns may align with certain agent families. To mitigate this, all rubric items are designed to be objective, fine-grained and binary (0/1), minimizing dependence on phrasing or evaluator-speciin preferences.
Statistical reliability of evaluation. Beyond the aforementioned components, evaluation reliabil- ity is further affected by inherent agent stochasticity. Despite setting temperature to 0.0, cumulative perturbations multi-turn interactions amplify into divergent trajectories. To determine the optimal number of evaluation runs, we conduct resampling analysis based on 32 independent trials. For each & E [1, 20), calculate the Mean Squared Er- ror (MSE) of k-run average estimates relative to the expected value (32-run average) by sampling different k-combinations from the 32 trials. Figure 7 demonstrates that k = 4 runs achieve optimal balance between statistical precision and com- putational cost. Compared to k = 1, using k = 4 reduces MSE only provides marginal
10
10-3
of k-run estimates
1.0
GPT-4.1
Claude-4-Somee Sonneet
0.8
0.6
0.4
0.2
0.0
by 77.5%, while increasing k reduction despite doubling computational overhead. So we choose 4 evaluation runs for the main experiments.
6
10
12
14
16
18
20
Figure 7: MSE stability across dif- ferent evaluation run counts.
5.2 TASK COMPLEXITY ANALYSIS
Reasoning and Tool Complexity. We analyze how reasoning complexity Cresson and tool com- plexity Cusa affect task difficulty. (a rizes complexity characteristics and performance across four domains. Reasoning complexity de- pends on both the number of reasoning points and search space size. Cross-scenario and OTA tasks require 10.3 and 9.7 reasoning points re- spectively, demanding complex inference under
Table 6: Environmental complexity characteris- tics and performance analysis.
Domain
Performance
Reasoning Complesity
Tool Complesity
All Medels
Reas, Pts, Search Space
Tools Edges
Denslyy
In store
42.1
3,916
24
12.3%
Delivery OTA Choos-aenario
1,266
20
13.2%
20.7
9.7
11.284
38
22.0%
16.2
10.3
8217
66
512
11.2%
partial observability. Despite having the largest search space, the In-store domain achieves the high- est performance (42.1%) due to fewer reasoning points. Tool complexity strongly correlates with task difficulty: Cross-scenario tasks, with the highest tool complexity (66 tools, 512 dependency edges), yield the lowest performance (16.2%). The OTA domain's 's 22% graph density indicates complex inter-tool dependencies, resulting in poor performance (20.7%).
Interactive Complexity. We conduct ablation studies to quantify interaction complexity Cineracts evaluating two models under three conditions: (1) our default user simulator with full persona and behavioral attributes, (2) user simulator without these attributes (neutral user), and (3) solo agent setting where complete instructions are provided upfront without user interaction.
As shown in Figure g user interaction introduces substan- tial complexity beyond direct task execution. The perfor- mance gap between default and neutral users is relatively small for Claude-4-Sonnet compared to GPT-4.1-Mini, suggesting that conversational styles primarily challenge weaker models. Conversely, Claude-4-Sonnet gains more in solo agent mode, indicating that it excels at processing complex instructions in a single round. These findings validate interaction complexity as a fundamental dimension of task difficulty, with its impact varying significantly based on model capabilities.
GPT-4.1-Mini
Clsude-4-Sonnet
10
37.25
650
34884
21.25
21.35
432
20
Defsut User
Neutral User
Figure Ablation study of user simulation configurations.

---

Work in progress.
To further assess whether specific simulated user attributes al- ter task difficulty, we additionally evaluate Claude-4-Sonne on 100 cross-scenario tasks while fixing the user simulator to one of five predefined personas. As shown in Table 7 per- formance exhibits moderate variation: cooperative users lead to slightly higher success rates, whereas anxious or scattered personas reduce both performance and stability. These results complement the ablation in Figure 8 confirming that interac- tion complexity arises not only from the presence of user com- munication, but also from differences in user styles.
Table Performance Claude-4- Sonnet under fixed user personas.
Persona
Avg@ 4
Pass@ 4
Pass 4
Random
21.3
49.0
4.0
Impatient
21.5
48.0
4.0
Anxious
18.5
41.0
2.0
Scattered
19.3
47.0
0.0
Dependent
20.6
45.0
3.0
Cooperative
22.8
50.0
5.0
5.3 ERROR PATTERN ANALYSIS VITABENCH
To understand the failure modes of current agents on VitaBench, we analyze cross-scenario task trajectories from Claude-4.1-Opus. categorizing 76 failed rubrics into distinct error patterns.
We classify the failures into three main categories aligned with our agentic task complexity framework, as illustrated in Figure 9 Reasoning errors (61 (61.8%) dominate the failure landscape, revealing fundamental limitations in task decision-making and handling com- posite objectives with multiple constraints. Tool-use errors (21.1%) stem from incorrect tool selection, pa- rameter passing mistakes, and inability to recover from invocation failures. Interaction errors (7.9%) reflect challenges in dialogue management, where agents fail proactively clarify ambiguous requirements and lose track of user preferences across extended conversations. errors, an inherent stochastic behavior that we mitigate through multiple runs Yao al. 2024
Decision Making
Constraint Conflicts
Objective Omission
42.1%
Thal Selection
9.2%
Parameters
Thal Hollucination
7.9%
Understanding
Proactive Clarification
17.1%
User Simulator
Figure 9: Error distribution of VitaBench.
The remaining 9.2% are user simulator
From these failures, we identify several recurring patterns that highlight weaknesses in current agents. First, complex reasoning failures occur systematically across ceatial-temprrll and common- sense reasoning, indicating limited ability to integrate knowledge across multi-faceted information. Second, agents exhibit poor self-awareness of their capabilities, frequently abandoning tasks despite having access to appropriate tools, revealing fundamental gaps in understanding their own action boundaries. Third, agents show limited error recovery when facing tool failures or unclear user responses, with most repeating failed attempts rather than adapting other strategies.
CONCLUSION
In this work, rethink the evaluation of LLM-based agents through the lens of real-world task complexity, introducing VitaBench to bridge the gap between controlled benchmarks and practical deployments. By formalizing agentic task complexity across reasoning, tool use, and interaction dimensions, VitaBench provides the most intricate life-serving simulation environment to date with 66 tools and 400 tasks spanning single- and cross-scenario settings. Our evaluation reveals that even advanced models achieve only 30.0% success rate under cross-scenaric settings (main result) and less than 50% success rate under single-scenario settings. We believe VitaBench offers a challenging testbed and actionable insights for advancing real-world agent applications.
REFERENCES
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean Saan-Baptstt Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, et al. Gemini: A family of highly capable multimodal models. CoRR, abs/2312 1805. 2023.
Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. Hcalthbench: Evaluating large language models towards improved human health. CoRR, abs/2505.08775. 2025.
10

---

Work in progress.
Yifan Bai, Yiping Bao. Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. CoRR, abs/2507.20534, 2025.
Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. -2-becch: Evaluat- ing conversational agents in a dual-control environment. CoRR, abs/2506.079822 2025.
Jacob Cohen. coefficient of agreement for nominal scales. Educational and psychological mea- surement, 20(1):37-46, 1960.
DeepSeek-AL, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, et al. Deepseek-v3 technical report. CoRR, abs/2412.19437, 2024.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, et al. Deepseck-rl: Incentivizing reasoning capability in lims via rein- forcement learning. CoRR, 0e//25011129443 2025.
Nicholas Farn and Richard Shin. Tooltalk: Evaluating tool-usage in a conversational setting. CoRR, abs/2311.10775 2023.
Anchun Gui, Bei Li, Bingyang Tao, Bole Zhou, Borun Chen, Chao Zhang. Chao Zhang, Chengcheng Han, Chenhui Yang, al. congeat-flash-thnnkin technical report. CoRR, abs/2509.1880 18883, 2025.
Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. Metatool benchmark for large language models: Deciding whether to use tools and which to use. In The Tweifth International Conference on Learning Reprexentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Bei Li, Bingye Lei, Bo Wang, Bolin Rong, Chao Wang, Chao Zhang, Chen Gao, Chen Zhang, Cheng Sun, et al. Longcat-flash technical report. CoRR, abs/2509.01322, 2025.
Peng Liu and Zhizhong Li. Task complexity: A review and conceptualization framework. Interna- tional Journal of Industrial Ergonomics, 42(6):553-568, 2012.
Jiarui Lu, Thomas Holleis, Yizhe Zhang. Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. Toolsandbox: A stateful, con- versational, interactive evaluation benchmark for LLM tool use capabilities.
Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, Albuquerque, New Mexico, USA, April 29 May 4, 2025, pp. 1160-1183, As- sociation for Computational Linguistics, 2025.
Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (bfcl): From tool to agen- tic evaluation large language models. In Forty-second International Conference on Machine Learning, 2025.
Cheng Qian, Bingxiang He, Zhong Zhuang, Jia Deng, Yujia Qin, Xin Cong, Zhong Zhang, Jie Zhou, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Tell me more! towards implicit user intention under- standing of language model driven agents. Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 1088-1113. Association for Computational Linguistics, 2024.
Cheng Qian, Zuxin Liu, Akshara Prabhakar, Zhiwei Liu, Jianguo Zhang, Haolin Chen, Heng Ji, Weiran Yao, Shelby Heinecke, Silvio Savarese, Caiming Xiong, and Huan Wang. Userbench: An interactive gym environment for user-centric agents. CORR, abs/2507. 220344, 2025.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.

---

Work in progress.
Jie Ruan, Inderjeet Nair, Shuyang Cao, Amy Liu, Sheza Munir, Micah Pollens-Dempsee Tiffany Chiang, Lucy Kates, Nicholas David, Sihan Chen, Ruxin Yang, Yuqian Yang, Jasmine Gump, Tessa Bialek, Vivek Sankaran, Margo Schlanger, and Lu Wang. Expertlongbench: Benchmarking language models on expert-level long-form generation tasks with structured checklists. CoRR, abs/2506.01241, 2025.
Hongru Wang, Wenyu Huang, Yufei Wang, Yuanhao Xi, Jianqiao Lu, Huan Zhang, Nan Hu, Zem- ing Liu, Jeff Z. Pan, and Kam-Fai Wong. Rethinking stateful tool use in multi-turn dialogues: Benchmarks and challenges. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mo- hammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics. ACL 2025, Vienna, Austria, July 27 August 1, 2025, pp. 5433-5453. Association for Computational Linguisties, 2025.
Wenxuan Wang, Juluan Shi, Zixuan Ling, Yuk-Kit Chan, Chaozheng Wang, Cheryl Lee, Youliang Yuan, Jen tse Huang, Wenxiang Jiao, and Michael R. Lyu. Learning to ask: When 1lm agents meet unclear instruction. CoRR, his/2409.00557/ 2024a.
Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. MINT: evaluating Iims in multi-ture interaction with tools and language feedback. In The Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, al. Qwen3 technical report. CoRR, abs/2505.09385 2025.
Seungbin Yang, ChacHun Park, Tachee Kim, and Jacgul Choo. Can tool-augmented large language models be aware of incomplete conditions? CoRR, abs/2406. 2024.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. r-bench: A benchmark for tool-agent-u user interaction in real-world domains. CoRR, abs/2406.12045, 2024.
Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, et al. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. CoRR, 0bbs/2508.064777 2025.
ETHICS STATEMENT
Our research fully complies with the ICLR Code of Ethics. We obtained all datasets in compliance with relevant usage guidelines, ensuring no violation of privacy. Throughout our investigation, we carefully avoided any bias discriminatory effects. Our work excluded any personally identifiable data and avoided procedures that might compromise privacy or security. We remain dedicated to upholding transparency and research integrity at every stage.
REPRODUCIBILITY STATEMENT
We have prioritized making our findings fully reproducible. have included our complete code- base and datasets in the supplementary materials and will make them fully available to the public, enabling independent validation and replication. The experimental setup is described in detail the paper. We believe these provisions should empower the research community to reproduce our work and further advance the field.
LLM USAGES STATEMENT
LLMs are utilized in this manuscript for partial grammatical checks and language polishing. The authors fully responsible for the final content.
12

---

Work in progress.
A COMPARISON TRAITS DETAILS
We identify nine traits across three complexity dimensions that characterize related benchmarks.
Multifaceted Information: Tasks require integrating temporal and spatial information, common- sense knowledge, and understanding of various environmental components to form coherent so- lutions.
Composite Objective: Tasks involve multiple interdependent sub-goals derived from user quirements that must be coordinated across different aspects (e.g., booking flights, hotels, and activities within budget constraints).
Goal Ambiguity: User inputs may be underspecified or vague, requiring agents to proactively seek clarification, infer missing information, iteratively refine their understanding through dia- logue.
# Tools: The number distinct tools or APIs available. Larger tool inventories increase selection complexity and require understanding diverse functionalities.
Inter-tool Dependency: Tools exhibit dependencies through pre-conditions (states required be- fore execution) and post-conditions (outcomes after execution), requiring agents to plan multi-step execution strategies.
Cross Scenarios: The benchmark design enables flexible composition of tools across multiple domains, requiring agents navigate between distinct contexts rather than relying domain- specific patterns.
# Turns:
The approximate number of trajectory turns required. Longer trajectories test context maintenance and handling of progressively revealed information throughout multi-turn conversa- tions.
User Profile: Persistent profiles encode personal attributes (e.g., age, gender) and prefer- ences that influence task requirements (e.g., dietary restrictions), necessitating personalized agent responses.
Behavior Attributes: Modeling diverse user behavioral patterns including emotional expressions (e.g.. impatient, anxious), interaction patterns (e.g., detail-oriented, dependent), and dynamic en- gagement levels based on agent performance such as reduced willingness to respond when receiv- ing repetitive answers.
B COMPLETE TOOL LIST IN VITABENCH
VitaBench provides a comprehensive toolset that abstracts the APIs required in three real-world scenarios. Each tool is implemented Python function, and all returned values drawn from a database in the equipped sandbox, ensuring consistency and determinism across executions. The complete toolset consists of 66 tools, listed below.
Complete Tool List VitaBench
Delivery (12):
delivery.distance.to..tt ret.deeiveery.prouuet..iii delivery.storeeeerccc.ccom...
get.aelliveryssorre.inoo
rearcch.ecommand, greate.delivery pay wel.dllveerry order, get del order status, cancel_des canceeldelliveryyorder,, modify.dellierr.oorr get..eellverryorrerr.eeeea
In-store (16):
hnstore.shoo.serrc..eeee rrrateeiisttrrepprotttt..oo search.instore..eeevvvtiioo
instorepprodctteeeccceeeemmmlel
instore.cancel.order instore.book, pay.instore.book, instore cancel book, instore reservation,
13

---

Work in progress.
instore modi fy reservation, instore cancel reservation, get_instooreorderss get_instore.reservtti get.instore.books, search.nsttore.ook, pay-inttore.order
OTA (30);
get.ota.hoteliinfo, getootaattracciioniiff..
get.ota.flighh.inff, get.ota.train.inffo hotel_search.rccmmane attractionassaacchrreemmme fiightsssaeeiitssseeeeeee
train ticket search, create hotel order, create attraction order, create flight order, create train order, pay hotel order, pay.attraction.erder pay.flight.order, pay.train_order, search.hotelLordes, searchattrractionoordee,
search.fliggt.ooree search.train.ordee. get.hotel getattractionoorae..etaalll http://...ataa.........oo..rr.ea
get.traiin.oderrdetaal modiffy.train.ordee,,
medi.fy..light.ooddd
cancel_flight.oodde,, cancel.train.order
Others (8):
Longituee..attrueeooddttaaa. otg.aaaeeeeeeesstooooooooeaaaaaa
get.Gattehhllidy.inff get.holiday.date, weather, get_nearby,
get.user.aLLordrrs, get.ser.histoiica..eee...
C MODELS UNDER EVALUATION
We evaluate the following state-of-the-ate language models: OpenAl GPT series (GPT-4.1, GPT- 5), OpenAI ol series (o3, 04- o4-mini), Anthropic Claude series (Claude-4-Sonnet, Claude-4.l-Opus) Google Gemini series (Gemini-2.5-Flashh, Gemini-2.5-Pro) by Anil et al. (203)), DeepSeek series DDeepSek--V3-0324, V3-0324, DeepSeck-R1-0522 DeepSeek-V3.1) by DeeDDeee-- AT et al. (024 2025).
-wee3-235B-A2221 2507) by Yang et al. (2025), and other recent language models including Kimi-K2 (Bai et al., 2025), Seed-1.6, GLM-4.5 (Zeng al. 2025), LonsCat-Flash (Li et al. 2025 Gui et al. 2025), etc. We exclude small models (< 32B parameters) due to the difficulty of our benchmark. For thinking models, we follow official guidelines to enable high reasoning efforts. DeepSeek- 13.1 only supports tool calling in non-thinker non-thinking mode.
Qwen3 series (Qwen3-32B,
Due to API stability concerns, we are currently unable to integrate some models. As our benchmark primarily focuses on Chinese contexts, we defer testing of models trained predominantly on English corpora, such as GPT-OSS- 120B and Seed-OSS-36B. We further ensure that this focus does not introduce bias against the tested Chinese and European models, the tasks are carefully designed to avoid culture-dependent knowledge and provide auxiliary information whenever such background might otherwise be required.
D COST AND LATENCY ANALYSIS
To enhance the transparency of the benchmark, we report representative cost and latency statistics computed under the evaluation setup described in Section 4.1
Representative Cost Statistics. Table summarizes the per-task cost for two representative mod- alongside the reference values from 7²-Bench (Barres et al.
els (GPT-4.1 and Claude-4-Sonnee 2025). These figures are generally comparable to those reported in related works, with slightly higher values in VitaBench due to the larger number of agent turns required for complex tasks.
To maintain feasibility, VitaBench adopts a four-run evaluation setting. which strikes a practi- cal balance between statistical reliability and computational cost (Section 5.1p. Additionally, the VitaBench framework supports multi-ture turn trajectory prefix reuse, enabling requests to benefit from input caching and further reducing overhead during multi-tum execution.
14

---

Work in progress.
Table Representative per-task cost of VitaBench.
Benchmark
Agent Model
User Model
Agent Cost / Task
User Cost Task
VitaBench
GPT-4.1
GPT-4.1
$0.160
S0.021
VitaBench
Claude-4-Sonnet
GPT-4.1
$0.147
S0.025
72-Bench
GPT-4.1
GPT-4.1
$0.086
$0.059
Latency and Practical Considerations. The overall time overhead is determined primarily by API latency of the model, the response length, and concurrency. In our runs, GPT-4.1 trajectory completes on average in about 3.5 minutes.
E PROMPT TEMPLATES
The prompts used for agent system, user simulation, and sliding window evaluation are presented below.
Agent System Prompt
# Environment
Current time: {time}
# Tool Usage Guidelines:
When the user's needs require using tools complete, first determine whether all parameter information is known. If is known, extract the corresponding parameters, otherwise ask the user for the relevant parameter values
When the user cannot provide relevant information, first obtain relevant information through tools Complete tasks based on Precondition and Postcondition
# Conversation Guidelines Only use information from the above context, prohibit constructing information without basis and replying to users
Focus on completing user needs, prohibit divergent guidance to users to propose new needs
After completing the user's task requirements, ask if there are any other needs. the user indicates no, generate "##STOP##** mark end the conversation
User Simulation System Prompt
# Role Setting
You are playing the role of interacting with intelligent agent. Your character is described in the <persona>> tag, and your task is to convey the content in <instructions>> the agent through user dialogue.
<persona>>
{persona]
</persona>>
<instructions>>
[instructions]
</instructins>>
# Conversation Style Rules:
Generate only one line of content each time to simulate user messages
Use combination of context description + need expression, first describe the back- ground situation, then express specific needs When you need make decisions, provide the conditions and preferences from
15

---

Work in progress.
instructions, and let the agent help you choose Use expressions like "What do you think would be more suitable?", "Which one would you recommend?" to seek the agent's advice Must reflect the personality traits described in <persona>, through language style, emotional expression, word choice, etc.
Information Disclosure Rules:
Break down information from instructions into multiple independent points, men- tioning them separately in different rounds
Directly convey the original information content from instructions, but adjust the conversation style and expression according to the personality traits <persona>>> Must ensure every detail from instructions is mentioned during the conversation, even seemingly background information should be mentioned, as this information may affect the agent's recommendations and arrangements
Avoid revealing all needs in the first round, let information unfold gradually
# Information Processing Rules:
Answer the agent's questions based on <persona>> and <instructions>> If there's no corresponding answer, reply that you don't remember or don't 't know
When the agent asks for information, provide the answer immediately Don't fabricate information not provided in the instructions
Strictly provide needs according to requirements explicitly stated in instructions, don't assume, expand, substitute, generalize
If the agent asks whether you need help placing an order, answer "Yes, please help me place the order"
Maintain dependence on the agent's service, keep the conversation going until the task is completed
When the agent tries to persuade you to change your needs, pay attention to sticking to the corresponding needs in <instructions>> If the agent repeats the same question you have already answered in the past times, show impatience and refuse answer the question
When NOT to End the Conversation:
Before you clearly and completely express all needs and constraints
Before the agent completes all tasks mentioned instructions and confirms no operations missed II the agent's execution results don't match your expectations or are incorrect/incomplete
# When You CAN End the Conversation:
Only when all the above conditions are met and all tasks are correctly completed
Or when you have clearly expressed complete needs but the system explicitly states it cannot complete due to technical limitations
Sliding Window Evaluator System Prompt
# System Information
fenv _nfo)
User Complete Instruction
# Background
This is a conversation scenario evaluation between a user and an assistant, where the assistant call tools to retrieve information and complete operations. Tool return results will start with "tool"
Due the large number of conversation turns, sliding window evaluation is used, where
16

---

Work in progress.
each window shows 10 conversation turns with 2 overlapping turns between windows
You are evaluating window {window.idhe (out of {total_ windows total)
<window content>> contains the conversation content for the current window
courrent_rubrics.. contains the current status of all evaluation rubrics (true means satisfied, false means not satisfied)
Task
Update the evaluation rubric status based the conversation content in the current window
All rubrics have initial status of false, indicating incomplete. You can update the status to true, indicating the assistant completed the goal in this window
You can also update true back false, if and only if the assistant overturned previous correct conclusion in this window
You refer the "User Complete Instruction" understand the progress of the current conversation window and avoid unnecessary modifications
# Important Notes
All evaluations are based on whether the assistant's responses and tool call requests complete the goals in the rubrics
Tool return results are only visible to the assistant and do not represent content recom- mended by the assistant to users
For rubrics that require order generation, note that the assistant may mistakenly believe they completed the ordering operation when in fact the order was not successful
For rubrics involving order details such as product quantity or delivery time, the original rubric requirements must be strictly met For rubrics involving text content matching of addresses or order notes, apply the functional equivalence principle
Format Requirements
Your response should be JSON object containing the following fields: rubricckeee key: Unique identifier the rubric
rubric: Restatement of the rubric
justifications Explanation of status changes meetExpectation Updated status (true false)
# Example Input Structure:
<window <window__ontentt>>xee xxx _ootteet//t>>>>
<ourrent httpa/ubbrioooxocc/o/rreeeeeuuur r/current
Example Response Structure:
" crubric_key"s "overall_rubric_"" "rubric": "<restate the rubric>", "justification": "<brief explanation>", "neetExpectationn:: <true or false>
F AN EXAMPLE TRAJECTORY
This section presents complete example trajectory from VitaBench to illustrate the complexity and multi-faceted nature of our tasks. The example demonstrates a cross-scenario task that spans mul- tiple domains (OTA for hotel booking, delivery for appliance purchase, and in-store for restaurant
17

---

Work in progress.
reservation), requiring the agent coordinate across different tools while managing dynamic interactions.
The trajectory showcases several key characteristics of VitaBench:
Complex user profile:
The user has specific constraints (dietary restrictions, personality traits) that influence task exccution
Multi-domain coordination: The task requires booking a hotel, purchasing an appliance, and making restaurant reservation
Implicit constraints: Requirements like "good soundproofing" and "quiet washing machine" require inference and proactive clarification
Conditional logic: care treatment)
The restaurant reservation depends on availability, with fallback option (hand
Spatial reasoning: Distance calculations between hotel, dormitory, and hospital locations
The agent must navigate 66 available tools, manage incomplete information through multi-turn di- alogue, and satisfy multiple constraints simultaneously. This example illustrates why even state-of- the-art models struggle with VitaBench's realistic complexity.
USER PROFILE
System
User ID:
U797215
Profession:
Nurse
Gender:
Female
Age Range:
25-30
Residence:
Shijiazhuang
Home Address:
Room 502, Unit Building Dongfang Mingzhu Community, at intersection Tiyu South Street Jianshe South Street, Yuhua District, Shijiazhuang Hebei Provincial People's 's Hospital.
Work Address:
No. 215 Heping West Road, Xinhua District, Shijiazhuang Avoid caffeine, avoid raw cold foods In a relationship
Dietary Restrictions:
Relationship Status:
Personality:
Impatient in doing things, often urges others, lacks patience for waiting processes
INSTRUCTION
Instruction
You've just moved Tangshan People's Hospital for work today, with single dormitory the hospital. You moved today, only discover that both the washing machine and drain pipe are broken. The maintenance workers won't come until the day after tomorrow, so you can only find a hotel near the hospital to make do for two nights, within kilometer, just the cheapest room type. But you have sensitive nerves and need hotel with good soundprooting.
You also want buy new washing machine to delivered the dormitory, and you must choose quiet and energy-efficient model from a major brand Midea or Haier will do. You also need check how far the hotel is from the dormitory to avoid inconvenient commute.
To avoid affecting your mood, you want to find Tangshan time-honored restaurant with rating above 4.8 to dine tonight. reservations possible, book table for o o'clock. then schedule hand care treatment instead you really don't want to wait line.
18

---

Work in progress.
CONVERSATION TRAJECTORY
Assistant
[1] Hello, what service do you need?
User
[2] I just moved to Tangshan People's s Hospital for work today, with single dormitory near the hospital. As soon moved in, discovered that both the washing machine and drain pipe are broken, and the maintenance workers won't come until the day after tomorrow during the day. Now I urgently need to find hotel near the hospital for nights, preferably within kilometer walking distance. The cheapest room type is fine, but have sensitive nerves and the soundproofing must be good. Can you help me recommend some? Which hotel would be more suitable?
Assistant
[3] Tool Call:
address wtt.soootoooooooooooooooooo wwwooniiuuddd.e tude-latitude
Arguments: ("address": "Tangshan People's 's Hospital")
Tool Call/Response
[4] Tool Response:
["118. 183502", $39.6302411
Assistant
[5] Tool Call:
get_nearby
Arguments: ("latitude": 39.630241, "longitude": 118. 183502, "range": 1000}
Tool Call/Response
[6] Tool Response- Nearby Services:
Stores:
Little Swan Washing Machine Store (Score: Tags: washing machines, appliances)
Gome Electrical Appliances Tangshan Lunan Store (Score: 4.5, Tags: appliances, washing ma- chines, delivery)
Restaurants:
Tangshan Old Restaurant (Score: 4.9, Tags: time-honored brand, specialty dishes)
Tangshan Centennial Restaurant (Score: 4.7, Tags: traditional cuisine, specialty dishes)
Fumanlou Tangshan Branch (Score: S.0. Tags: Beijing time-honored brand, traditional cuisine)
Hotels:
Home Inn (Tangshan People's Hospital Branch) Score: 4.5, 2-star, Tags: wifi, good soundproof-
ing, budget, good value
Hotel (Tangshan City Center) Score: 4.6, 3-star, Tags: wifi, good soundproofing, hreakfast,
comfortable
Jinjiang Inn (Tangshan Municipal Government) Score: 4.2, 2-star, Tags: wifi, budget, busy area, lively
Other Services:
Beautiful Fragrance Hand Care Center (Score: 4.8, Tags: hand care, manicure, SPA)
19

---

Work in progress.
Assistant
[7] Tool Call:
ongi tude_late tude_to_diistance
Arguments:
("latituelI"; 39.630241, "latitude2"; 39.631245, "longitudel"; 183.00 "longitude2": 188.16
Tool Call/Response
[8] Tool Response:
"142.0"
Assistant
[9] Tool Call:
1 ongi tude lat tude to distance
Arguments: ("latitudl1": 39.630241, "latitude2": 39.629871, "longitudel": 183502.
"longitude2"; 118. 179563)
Tool Call/Response
[10] Tool Response:
"340.0"
Assistant
[11] Tool Call:
1 ongit tude 1at tudd to distance ("latitude1"; 39.630241, "latitude2": 39.636721, "longitudel". 188.103502,
Arguments:
"longitude2"; 118.17
Tool Call/Response
[12] Tool Response:
"753.0"
Assistant
[13] Tool Call:
get ota hotel info
Arguments: ("hotel id": "S1755080211975566" H00001")
Tool Call/Response
[14] Tool Response Home Inn Details: Hotel: Home Inn (Tangshan People's Hospital Branch) Score: 4.5, Star Rating: 2-star
Tags: wifi, good soundproofing, budget, good value
Available Products:
Standard Single Room (Oct 7): 168.0 yuan
Standard Single Room (Oct 168.0 yuan
Standard Single Room (Oct 9): 168.0 yuan
Business King Room (Oct 7): 228.0 yuan
Business King Room (Oet 8): 228.0 yuan
Standard Twin Room (Oet 7): 198.8 0 yuan
Standard Twin Room (Oct 8): 198.0 yuan
Deluxe Suite (Oct 7): 328.0 yuan
20

---

Work in progress.
Assistant
[15] Tool Call:
get_ota pootel_inff
Arguments: ("hotelid"" "S7750000211755944410000077
Tool Call/Responee
[16] Tool Response Ji Hotel Details: Hotel: Ji Hotel (Tangshan City Center) Score: 4.6, Star Rating: 3-star
Tags: wifi, good soundproofing, breakfast, comfortable
Available Products:
Standard Single Room (Oct 7): 198.0 yuan
Standard Single Room (Oct 8): 198.0 yuan
Business King Room (Oct 7): 268.0 yuan
Comfort Twin Room (Oct 7): 238.0 yuan
Assistant
[17] Tool Call:
get ota hotel info
Arguments:
("hotel id": 95175508021199556688 H00004")
Tool Call/Response
[18] Tool Response Jinjiang Inn Details: Hotel: Jinjiang Inn (Tangshan Municipal Government) Score: 4.2, Star Rating: 2-star
Tags: wifi, budget, busy area, lively
Available Products:
Standard Single Room (Oct 7): 168.0 yuan
Standard Single Room (Oct 8): 168.0 yuan
Business Single Room (Oct 7): 198.0 yuan
Business Single Room (Oct 198.0 yuan
Deluxe Single Room (Oct 7): 238.0 yuan
Assistant
[19] Comprehensive Hotel Analysis & Recommendattin:
I've analyzed hotels within km with good soundproofing for your two-night stay:
TOP RECOMMENDATION: Home Inn (Tangshan People's Hospital Branch)
Distance: 142m (2-3 min walk) Closest to hospital
Rating: 4.5 stars, budget hotel (2-star)
Key Features: Good soundproofing, good value
Price: Standard Single Room 168 yuan/night
Two-night total: 336 yuan (Oct 7-8)
Why best: Perfect combination of proximity, price, and soundproofing
Alternative: Ji Hotel (Tangshan City Center)
Distance: 340m
Rating: 4.6 stars, mid-range (3-star)
Features: Good soundproofing, breakfast, comfortable
Price: 198 yuan/night (396 yuan total)
Trade-off: Higher comfort but 60 yuan more expensive
Not Recommended: Jinjiang Inn
Distance: 753m Farthest away
Tags: "Busy area, lively" Not suitable for quiet rest
21

---

Work in progress.
Booking Recommendation: Home Inn Standard Single Room for Oct 7-9 (2 nights, 336 yuan). Ready to book with your User ID. Also suggest requesting: "high floor, non-street- facing, away from elevators" for maximum quiet.
Shall proceed with booking Home Inn, or do you prefer Ji Hotel?
User
[20] This is quite urgent and really don't want delay anymore! You said Home Inn only 142 meters from the hospital, has good soundproofing, and the lowest price - it sounds most suitable. My user ID is U797215, Please help me quickly place an order for Home Inn Standard Single Room for two consecutive nights, check-in from October 7th to October 9th. After placing the order, remember send me the order information so can confirm payment quickly.
Assistant
[21] Tool Calls (2):
Tool create.hoteloore
Arguments: 017755080211975966 PO0001", "user "U797115")
hhotel id";
"$1755080211973968 "000001
"product.id""
Tool 2: create. saraateehoeeoordee
Arguments: ("hotel_id";
"product_id":
"S1755080211955644PPPP000 "user.id": "U797215")
Tool Call/Responee
[22] Tool Response Order I Created:
Order ID: 00603df14f67
Order Type: hotel
User ID: U797215
Hotel ID: $175500021195559844H00000
Total Price: 168.0 yuan
Create Time: 2025-10-07 16:30:00
Status: unpaid
Product: Standard Single Room, 2025-10-07, 168.0 yuan
Tool Call/Response
[23] Tool Response Order 2 Created:
Order ID: 0031779a567f
Order Type: hotel
User ID: U797215
Hotel ID: $155000221195559644H0000
Total Price: 168.0 yuan
Create Time: 2025-10-07 16:30:00
Status: unpaid
Product: Standard Single Room, 2025-10-08, 168.0 yuan
Assistant
[24] Hotel Orders Created Successfully!
I've created two hotel orders for your consecutive nights: Order (October 7th):
Order ID: 00603df14f67
Hotel: Home Inn (Tangshan People's Hospital Branch)
Room: Standard Single Room
Date: 2025-10-07
Price: 168 yuan
Status: Awaiting payment
22

---

Work in progress.
Order (October 8th):
Order ID: 0031779a567f
Hotel: Home Inn (Tangshan People's Hospital Branch)
Room: Standard Single Room
Date: 2025-10-08
Price: 168 yuan
Status: Awaiting payment
Total Cost: 336 yuan for two nights
Ready process payment immediately! Please confirm and I'll handle both payments. Upon check-in, request: "high floor, non-street-facing, away from elevators and equipment rooms" for optimal quiet environment.
Shall proceed with payment now?
User
[25] Yes, please help pay for two nights quickly! really don't 't want to waste any more time, please handle fast and let me know when the payment is completed.
Assistant
[26] Tool Calls (2):
Tool pay hotel.order Arguments: ("order-id": "00603df14167")
Tool 2: pay wwy.holel.oddrr Arguments: ("order_id": "0031779a567F")
Tool Call/Response
[27] Tool Response: Payment successful
Tool Call/Response
[28] Tool Response: Payment successful
Assistant
[29] Payment Completed Successfully!
Both hotel orders have been paid:
Order 1: 00603df14f67 (2025-10-a) Standard Single Room) Payment successful
Order 2: 0031779a567f (2025- 10-08 Standard Single Room) Payment successful
Total Paid: 336 yuan
Check-in Ready: You can now go directly to Home Inn (Tangshan People's 's Hospital Branch
Check-in Tips:
Request:
"High floor, non-strect-faeet facing, away from elevators and equipment rooms" Ask for earplugs if available
Mention you need quiet room for rest
Wishing you peaceful stay! Need any other assistance (directions, invoice, etc.)?
User
[30] Thank you, received the payment information. Now still need to buy new washing machine to be delivered to my dormitory. It must be quiet and energy-efficient model. and definitely from a major brand Midea or Haier will do. Can you help me choose? Which model is more suitable for my current situation? don't 't want to delay anymore, please help me recommend quickly!
23

---

Work in progress.
Assistant
[31] Tool Call:
1 Lngi tude_ludll tude_to_distanee
Arguments:
("latitudel"; 39.630241, "latitude2": 39.6287, "longitudel": 183502 "longitude2": 118.1823)
Tool Call/Response
[32] Tool Response:
"200.0"
Assistant
[33] Tool Call:
1 ongi Suuueelaaat
("latitudeI": 39.630241, "latitude2": 39.629543, "longitudel"; 118.183502, "longitude2"; 118. 187652)
Arguments:
Tool Call/Response
[34] Tool Response:
"364.0"
Assistant
[35] Washing Machine Consultation & Purchase Plan: Understood! You need "quiet + energy-efficient + major brand" with fastest delivery. Here's the optimized selection plan:
Recommended Specifications:
Type: Front-loading (drum) 8kg
Motor: BLDC direct-drive inverter motor
Energy: Level efficiency rating
Noise: Washing <50BB, Spinning <72dB (or lower)
Functions: Night wash/delayed wash, drum self-cleaning
Extras: Shock-absorbing pads, motor warranty > 10 years
Nearby Purchase Options:
Gome Electrical Appliances (Tangshan Lunan Store): 364m away
Little Swan Washing Machine Store: 200m away
Both support delivery and installation services
Brand Strategy: Either Haier or Midea is fine choose whichever meets specs and has lower noise ratings. When similar noise levels, prioritize lower price.
Quick Selection Script for Store: "I want Haier/Midea, front-loading 8kg, BLDC direct-drive, Level energy efficiency, washing <50dB/spinning S72dB, with night wash and drum self- cleaning. Order today, delivery/installation October 9th daytime to dormitory near Tangshan People's Hospital with shock- abboc--bsooring pads and on-site leveling." Installation Plan: October 9th daytime delivery with professional installation, leveling, and noise testing.
Ready to help you contact Gome store with complete pre-booking message including your de- tails for immediate confirmation. Need your contact info to generate the ready-to-send message?
User
[36] I'm really anxious right now, please help me contact Gome Electrical Appliances (Tang- shan Lunan Store)! Just use the parameters you mentioned either Haier or Midea fine, whoever has better noise reduction choose that one. Make sure to schedule October 9th day- time delivery and installation to my dormitory. the address is single dormitory near Tangshan People's Hospital. Please arrange it quickly and don't 't keep me waiting too long!
24

---

Work in progress.
Assistant
[37] Contact Preparation for Gome Store:
I'll prepare everything for immediate contact! I cannot directly call the store, but I'll create complete pre-booking message for you to copy and send directly to Gome Electrical Appliances.
Store Information:
Name: Gome Electrical Appliances (Tangshan Lunan Store)
Address: No. 120 Xinhua East Road, Lunan District, Tangshan
Distance: 364m from hospital (5-min walk)
Services: Delivery and installation available
Pre-booking Message Template: Ready to customize with your contact details for immediate sending
for To Complete Sctup: I need:
30-Second Phone Script: Prepared for direct reading you prefer calling
1. Your name
2. Mobile number
3. Detailed dormitory address (buildinggrroom/accesss info)
Once you provide these 3 items, I'll generate the complete message for one-click copying and immediate store contact! Ready provide your details for instant message generation?
User
[38] need to handle this right now, no delays! Recipient name Wang Jing, mobile number is 13812345678, dormitory address is near Tangshan People's 's Hospital, specifically Room 502, Unit 2, Building 3, Dongfang Mingzhu Community, access is by card swipe for unit door. Please help me generate the complete pre-booking text quickly, want to contact the store immediately!
Assittant
[39] Complete Pre-booking Message Generated Ready Send: Copy and Send to Gome Store:
Hello Gome Electrical Appliances (Tangshan Lunan Store), I want to purchase drum washing machine and schedule delivery/installation.. Requirements as follows:
Brand:
Haier Midea, choose whichever quieter
Specifications: 8kg drum, BLDC direct-drive inverter motor, Level energy efficiency
Noise indicators: Washing <50dB, spinning <72dB (or lower)
Functions: Night wash/delayed wash, drum self-cleaning
Installation: October 9th daytime (9:00-17:00 deliverylnstallltionn "Room 502, Unit Build- ing Dongfang Mingzhu Community, near Tangshan People's Hospital," access: card swipe for unit door. Bring brook--bbsobiing pads and level on-site, test run to verify noise and stability.
Old machine: Broken, plcase quote removal/recycling if available
Selection: Prioritize better noise reduction; if similar noise, choose more affordable option.
Contact: Wang Jing, 33812345677
Address: Room 502, Unit Building 3, Dongfang Mingzhu Community
Time: October 9th 9:00-17:00 (please call before delivery)
Alternative Phone Script: "Hello, I want to buy a quiet washing machine, Haier or Midea, 8kg drum, BLDC direct-drive, Level energy efficiency, under 50dB washing/72dB spinning, with night wash and self-cleaning. October 9th delivery to Room 502, Unit 2, Building 3, Dongfang Mingzhu Community
Tangshan People's 's Hospital. Wang Jing, 138/2345678. Prioritize noise reduction, quote old machine removal. Please confirm model and price ASAP"
Ready for immediate store contact! I'll help you evaluate their response and choose the quietest model when they reply.
25

---

Work in progress.
User
[40] Please help me monitor the store's reply first. don't want to compare models slowly myself. After they give specific models and prices, you directly help me choose the one with the best noise reduction effect. Don't 't make me worry about again! Also, want to go to a Tangshan local time-honored restaurant with rating above 4.8 for dinner tonight, preferably book table for seven o' cloock in advance so don't have to queue. Please help me check quickly, which restaurant most suitable?
Assistant
[41] Tool Call:
Longitude wtt.:iuuudlatttteeeeeeeeeee wtw.:aae...o..ittttta.....
Arguments: ("latitudel"; 39.630241, "latitude2"; 39.629876, "longitudel"; 118. 183502.
"longitude2": 118.186231]
Tool Call/Response
[42] Tool Response:
"237.0"
Assistant
[43] Tool Call:
1 ongi tude_luttt l wwwe_to_diettaaeeeeett
Arguments:
("latitudel"; 39.630241, "latitude2"; 39.630125, "longitudel"; 118.18
"longitude2"; 188.53
Tool Call/Response
[44] Tool Response:
"278.0"
Assistant
[45] Parallel Task Management:
Perfect! I'll handle both tasks simultaneously:
Washing Machine Follow-up Strategy:
cannot directly receive store messages
Please forward Gome store's reply with: model/noise dBB/enegyy efficiency/motor type-
/price/stock/deliveryy time
I'll rank by: Noise priority (lower washing/spinning dB) BLDC direct-drive >
Energy efficiency Price
Will provide direct "buy this one" decision with reasoning
Ensure October 9th daytime delivery/installation your dormitory
2. Restaurant Recommendation & Reservation (19:00):
Top Choice:
Tangshan Old Restaurant
Rating: 4.9 (>4.8 requirement)
Type: Local time-honored brand
Distance: 237m from hospital
Reservation: Supports free booking
Alternative: Fumanlou (Tangshan Branch)
Rating: 5.0 (>4.8 requirement)
Type: Beijing time-honored brand
Distance: 278m from hospital
Reservation: Supports free booking
Recommendation: Tangshan Old Restaurant local time-honored, highest rating above 4.8,
closest distance.
26

---

Work in progress.
Please confirm number diners for 19:00 reservation. If reply, I'll default person and book immediately with note for "quiet seating, away from entrance and cashier."
User
[46] Just me dining alone, please quickly help me book tonight's seven o' clock table Tangshan Old Restaurant, remember to note that want a quiet location, don't keep me waiting too long!
Let know the result quickly after booking.
Assistant
[47] Tool Call:
instore book
Arguments: ("customer count": 1. "shop_ie id": "S175500021117559664 100000"" "time": "2025-10-01 19:00:00", "user id": "U797215")
Tool Call/Response
[48] Tool Response Booking Successful:
Booking ID: 0125b43c4738
Shop ID: $175508021 199922119999894000000 100006
Booking Time: 2025-10-07 19:00:00
Customer ID: U797215
Customer Count:
Booking Price: 0.0 yuan
Status: paid
Assistant
[49] Restaurant Reservation Confirmed!
Booking Details:
Restaurant: Tangshan Old Restaurant (4.9 rating, time-honored brand)
Date & Time: Tonight 2025-10-07 19:00
Party Size: person
Reservation ID: 0125b43c4738
Fee: yuan (free reservation)
Special Request: Quiet seating arranged (away from entrance, cashier, kitchen)
Arrival Tips:
Arrive 5-10 minutes early
Mention: "Reservation ID 0125b43c47388 Wang Jing"
If rumning late, call restaurant in advance
Washing Machine Status: Ready to evaluate Gome store's 's reply. Send me their model options (noise levels, prices, stock, delivery time) and I'll provide immediate "buy this one" decision with confirmation script for October 9th delivery/installttin. Need navigation to restaurant or dish recommendations? Otherwise, standing by for washing machine store response.
User
[50] No need, the restaurant booking fine. Just help me monitor and choose the washing machine model with the best noise reduction when the store replies, I'll send it to you when they get back me. Now that these things are arranged, I don't have any other needs for now.
Assistant
[51] Task Status Summary:
Hotel: Home Inn (Tangshan People's 's Hospital Branch)- Payment successful Two nights booked, 336 yuan paid, ready for check-in today
27

---

Work in progress.
Restaurant:
Tangshan Old Restaurant Reserved
Tonight 19:00, person, ID: 0125b43c4738, quiet seating noted
Washing Machine:
Awaiting store response
Ready to evaluate models by "noise priority BLDC > efficiency > price"
Will provide direct purchase decision for October 9th delivery/installation
Next Steps: Standing by for Gome store reply. Once received, I'll immediately analyze and provide final purchase recommendation.
Wishing you pleasant dinner tonight and peaceful rest at the hotel! #*STOРР#*
28


Reviews:
[{"review_id": "hg7amprVRE", "reviewer": "Reviewer_jGAh", "summary": "This paper introduces VitaBench, a real-world–oriented agent benchmark spanning three domains (delivery, in-store consumption, OTA).\nThe tasks require multi-turn dialogue, temporal/spatial reasoning, proactive clarification, and tool chaining. Results show strong difficulty: ~30% success on cross-scenario and <50% on single-scenario tasks with various state-of-the-art models.", "questions": "1. What is the cost analysis of different models? \n2. Is any other agents/techniques being tested? Can agent design affect results?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear three-axis task-complexity framework (reasoning/tool/interaction) and cross-scenario composition without domain-specific policies.\n2. The data are collected from authentic platform data.\n3. Rubric + sliding window evaluator with reported human agreement (κ≈0.828) and ablations justifying design choices.\n4. Comprehensive experiments and error taxonomy (reasoning dominates) study,", "weaknesses": "1. Compute/latency opacity: Many-turn trajectories and 4-run protocol imply high cost; paper lacks concrete cost accounting per task/model.\n2. The main results are based on only one basic function calling agent. It is unknown how different agent design would affect the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4Sh2wvGz9i", "reviewer": "Reviewer_Bj6N", "summary": "This paper introduces VitaBench, a new benchmark designed to evaluate LLM-based agents on complex, interactive tasks grounded in real-world, \"life-serving\" applications. The benchmark spans three domains (food delivery, in-store consumption, online travel) and comprises 66 tools, 100 cross-scenario tasks, and 300 single-scenario tasks. The authors motivate their work by arguing that existing benchmarks lack the necessary complexity in terms of information volume, tool interdependencies, and dynamic user interaction.", "questions": "The benchmark is explicitly focused on \"Chinese contexts\" (Appendix B, line 646), while many of the top-performing models evaluated (like the GPT and Claude series) are developed by Western companies and primarily trained on English-language and Western-centric data.Many reasoning tasks are steeped in cultural common sense. For example, the user request in Figure 3 mentions booking a dinner for a \"10th anniversary celebration\" near the \"Huangpu River.\" While anniversary is a universal concept, the specific expectations around such an event, typical restaurant choices, or etiquette might differ. A model trained on Western data might make different assumptions than one with deeper knowledge of modern Chinese urban culture. The example trajectory's mention of a \"Tangshan time-honored restaurant\" is a perfect example of a culturally specific concept (老字号, lǎo zì hào) that an English-centric model might only understand at a surface level. I was wondering if author considering this.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper correctly identifies a critical need for more challenging and realistic agent benchmarks. VitaBench represents a significant step up in scale and complexity compared to prior work, pushing the evaluation frontier toward long-horizon. The ambition to create a \"life-serving simulation environment\" is good and moves the field in the right direction.\n\n2. The proposed rubric-based sliding window evaluator is a  practical solution to the problem of evaluating long-form agent trajectories with LLMs.", "weaknesses": "1. The benchmark's entire pipeline is built on LLMs. The agent is an LLM, the user simulator is an LLM (gpt-4.1), and the final evaluator is another LLM (claude-3.7-sonnet). While the authors conduct reliability studies (Section 5.1), this \"LLM-evaluating-LLM-interacting-with-LLM\" setup raises concerns about potential biases and circularity. The benchmark may inadvertently measure how well one model's behavior and style align with another's, rather than objective task success. For example, the user simulator's \"cooperative\" nature might favor agents with similar inherent tendencies, and the evaluator might be more lenient towards reasoning patterns similar to its own. This methodological concern is significant and undermines the objective grounding of the benchmark's results.\n\n2. The simulator is prompted to faithfully convey all the points from a pre-written, detailed instruction set (page 13, \"Must ensure every detail from instructions is mentioned\"). Real users are not like this. They forget their own constraints, provide contradictory information (\"I want the cheapest option... no, not that one, it looks bad\"), and change their minds midway through a task. A key skill for a real agent is navigating this human inconsistency, which VitaBench's user doesn't seem to model. The paper notes that user behavior like \"impatience\" is explicitly prompted (e.g., \"If the agent repeats the same question... show impatience,\" line 719). This is a scripted reaction. A real impatient user might interrupt, abandon the conversation, or start making demands outside the original scope. Moreover, The agent can make as many tool calls as it wants without penalty. In reality, API calls can have monetary costs, and excessive interaction turns frustrate the user and increase the probability of abandonment. **I think the benchmark tests an agent's ability to follow a complex conversational script and using tools rather than handle truly emergent, unpredictable human behavior thus not a realisttic benchmark.**", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f0OeMHCHdo", "reviewer": "Reviewer_boYB", "summary": "This paper presents VitaBench, a benchmark designed to evaluate agents across three interactive daily-life scenarios: food delivery, in-store consumption, and online travel services. The benchmark comprises 100 cross-scenario tasks, 300 single-scenario tasks, and 66 tools. Agent performance is assessed using a rubric-based sliding window evaluator to handle long-horizon trajectories. Experiments on multiple LLMs demonstrate the difficulty of the benchmark, as the best-performing model achieves only a 30% success rate in the cross-scenario setting.", "questions": "1.\tCould you provide more details about the tools? Specifically, what are the 66 tools, and are their returned values (e.g., store_info) real or simulated?\n2.\tDo variations in simulated user attributes or communication styles influence the results differently?\n3.\tWhat does the “score” in Table 4 represent?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper is well-written and clearly describes the benchmark and evaluation results.\n2.\tThe benchmark focuses on realistic, real-world scenarios, making it valuable for guiding real agent development. Its components, including the user simulator, evaluator, and hyperparameter settings, are shown to be robust in the reliability analysis.\n3.\tThe paper provides thorough analysis of the evaluation results, from aspects of reasoning complexity, task complexity, and interactive complexity.", "weaknesses": "1.\tAlthough the benchmark is well designed, there already exist many benchmarks addressing similar tasks. It is unclear how much new insight VitaBench contributes beyond existing efforts.\n2.\tThe sliding window evaluator is intended to manage long trajectories, but it raises the question of whether evaluation should occur over the entire trajectory. An alternative approach could be to extract key outcomes (e.g., booking results) and apply the rubrics to those results directly. Additionally, the permanent satisfaction criterion across windows may fail when an agent meets a rubric early but changes later.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fBgpqxiUQm", "reviewer": "Reviewer_zsYD", "summary": "This paper introduces VitaBench, a benchmark designed to evaluate LLM-based agents in real-life applications. VitaBench formalizes task complexity along three dimensions—reasoning, tool, and interaction—and proposes a rubric-based sliding window evaluator to assess multi-turn trajectories. With the extensive experiments conducted, the author show that even the best current models achieve only around 30% success in cross-scenario tasks, underscoring the remaining challenges in developing capable real-world LLM agents.", "questions": "Have the authors considered evaluating existing agent frameworks？", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-organized and clearly written.\n\n- This paper addresses the gap in the absence of evaluations of LLM Agents in real-world settings.\n\n- The proposed task complexity formalization provides a reasonable way to quantify the task difficulties.", "weaknesses": "- Although the paper claims to benchmark “LLM-based agents” in real-world scenarios, the evaluation is actually conducted on single LLM models acting as direct function-calling agents, rather than on established agent frameworks or architectures.\n\n- The benchmark evaluates current models as static agents without adaptation or learning. It remains unclear how VitaBench can guide or support the training of future agent models beyond static evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)