TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use
Authors: Pengfei He
Zhenwei Dai
Bing He
Hui Liu
Xianfeng Tang
Hanqing Lu
Juanhui Li
Jiayuan Ding
Subhabrata Mukherjee
Suhang Wang
Yue Xing
Jiliang Tang
Benoit Dumoulin
Venue: ICLR 2026 Poster (2026)
Keywords: LLM agent
tool usage
benchmark
Primary area: datasets and benchmarks

Abstract: Large language model (LLM)-based agents increasingly rely on tool use to complete real-world tasks. While existing works evaluate the LLMs' tool use capability, they largely focus on the final answers yet overlook the detailed tool usage trajectory, i.e., whether tools are selected, parameterized, and ordered correctly. We introduce TRAJECT-Bench, a trajectory-aware benchmark to comprehensively evaluate LLMs' tool use capability through diverse tasks with fine-grained evaluation metrics. TRAJECT-Bench pairs high-fidelity, executable tools across practical domains with tasks grounded in production-style APIs, and synthesizes trajectories that vary in breadth (parallel calls) and depth (interdependent chains). Besides final accuracy, TRAJECT-Bench also reports trajectory-level diagnostics, including tool selection and argument correctness, and dependency/order satisfaction. Analyses reveal failure modes such as similar tool confusion and parameter-blind selection, and scaling behavior with tool diversity and trajectory length where the bottleneck of transiting from short to mid-length trajectories is revealed, offering actionable guidance for LLMs' tool use.

Published as a conference paper at ICLR 2026
TRAJECT-BENCHE TRAJECTORY-AWARE BENCH- MARK FOR EVALUATING AGENTIC TOOL USE
Pengfei He', Zhenwei Dai Bing He?, Hui Liu", Xianfeng Tang?, Hanqing Lu², Juanhui Li², Jiayuan Ding', Subhabrata Mukherjee', Suhang Wang', Yue Xing', Jiliang Tang', Benoit Dumoulin Michigan State University Amazon Inc. "Hippocratic Al *Penn State University
ABSTRACT
Large language model (LLM) agents increasingly rely on tool use to com- plete real-world tasks. While existing works evaluate the LLMs' tool use ca- pability, they largely focus on the final answers yet overlook the detailed tool usage trajectory, i.e., whether tools are selected, parameterized, and ordered cor- rectly. We introduce TRAJECT-Bench, a trajectory-awy aware benchmark to com- prehensively evaluate LLMs' tool use capability through diverse tasks with fine- grained evaluation metrics. TRAJECT-Bench pairs high-fidelity, executable tools across practical domains with tasks grounded in production-style APIs, and syn- thesizes trajectories that vary in breadth (parallel calls) and depth (interdependent chains). Besides final accuracy, TRAJECT- Bench also reports trajectory-level diagnostics, including tool selection and argument correctness, and dependen- cy/order satisfaction. Analyses reveal failure modes such as similar tool confusion and parameter-blind selection, and scaling behavior with tool diversity and trajec- tory length where the bottleneck of transiting from short to mid-length trajectories is revealed, offering actionable guidance for LLMs' tool use
INTRODUCTION
Large language models (LLMs) increasingly exhibit strong planning and reasoning abilitics (Wei et al.) 2022 Yao al. 2023a they break goals into subproblems, track intermediate states, and
arrange multi-step procedures (Huang et al., 2024, [Song et al.] (023). Together, these abilities po- sition LLMs the "brain" of agentic systems, guiding the systems to perform complex, real-world tasks (Wang et al.] 2024 He et al. 2024b Li et al.) 2024). Meanwhile, external tools, such as search engines (Jin et al. 2025), production APIs (Li al. (023a). and file/OS operations (Packer et al. (002)). function as the agent's 's "hands", extending LLM's 's abilities with precise computations, up-to-date information, and concrete actions. Tool-use has already powered agents across domains: travel agents (Chen et al. 2024 Singh et al. (2044) fuse flight and hotel APIs with visa and weather services to assemble feasible itinearrie and resolve constraints; and education agents (Chu et al.) 2025 Zhang et al.) 2024b) retrieve curriculum materials, generate adaptive exercises, grade against rubrics, and track progress across learning platforms.
Despite these developments, proper evaluations of LLMs' tool-use are essential: they can provide a complete picture of tool-use competence, expose failure modes, and point out concrete directions for improvement. Specifically, the capability select the right tool(s), determine the correct formats and values of input parameters, conduct multi-step tool utilization, and adapt to unseen tools, makes the core part of evaluating tool-use capability. Benchmarks have been created test these abilities. For example, Huang et al. (2023) focus on evaluating whether LLMS can determine when to call tools, and Qin al. (2023); Patil et al. (2024); Patil et al.) Zhuang al.] (2033 evaluate LLMs' capability in calling proper tools for solving complex queries.
However, significant gaps in tool-use evaluation still persist. First, tool-use trajectory complexity is comparatively underexplored: some existing suites rely on small or simulated tools (Zhuang et al.) 2023) and many only test short, low-depth tool trajectories (Qin et al. 2023). However, real agents
Cooe a http://wwieeooooooooo/ooo//o//// http://///iggiinnaaaoaee/oaaaaee
Benchh data available

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of previous work and TRAJECT-Beach
Practical
Large&diverse
Trajectory
Trajectory
Trajectry-awarr metries
Query difficully
Tool selection
Agentic
lools
tool
structure
scaling
evaluation
MetaTool Huane el al. 2023
API-Hank Tarraa 20243
X
X
X
ToolBench [0mn 2013
(Palol
(Panter at.
ToolQA
Zhirnng el al. 2023
X
X
X
V
TRAJEC --ench
can possess large tool sets and face complex user queries, so evaluation on a larger executable tool set and trajectories with more tools involved is still needed. Second, the complexity of real user queries is also underrepresented: existing benchmarks often consider straightforward prompts by including the API name directly in the prompt. In contrast, real-world agents may face user queries that consist of indirect language and implicit cues, and the agents need to infer both the choice of the tool and how to set the imputs. Third, most benchmarks still privilege final-answe metrics. For example, Qin et al. (2023) only provide pass rate and win rate the final answer, and Patil et al. heavily rely the overall accuracy. In this case, it is hard track the root cause of the incorrect final answer, which can be caused by various issues such as incorrect tool selection, disorders of tool-use, or incorrect parameterization. This oversight can obscure the evaluation of LLMs' tool-use capability and cannot differentiate it from other general reasoning capabilities, especially given the observation in (Roberts al., 2020 Qian et al. 2025b) that LLMs can solve the problem using internal knowledge even when the wrong tools are called.
To bridge these gaps, we present TRAJECT-Benche the LLMS tool-use capability via providing (1) tool-use trajectories of different complexities and (2) user queries of different levels of difficulties given the same tool-use trajectory. Meanwhile, we also include (3) evaluation metrics to evaluate the LLMs' tool-use capability from diverse perspectives. As summarized in Table I we are the first to provide all the comprehensive analysis perspectives for tool-use evaluation compared to existing benchmarks.
benchmark to comprehensively evaluate
During the data construction, to ensure the quality of the data and better align it with real scenarios, on the tool side, we select diverse suite of over 1,000 high-fidelity tools drawn from various real- world domains (c.g., finance, travel, music, etc.), exposed via production-style APIs. Given these tools, to construct (1) tool-use trajectories, that encode different trajectory structures and scales. We consider both parallel and sequential tool calling trajectory structures and trajectory scale of to 10+ tool counts. Moreover, for each trajectory, we provide (2) two semantically aligned queries of different query difticulties: a direct and explicit "simple" version and naturalistic and indirect "hard" version. Such a data construction procedure allows us to decompose the queries' difficulty into different perspectives and understand the corresponding specific weaknesses of LLMs.
further synthesize task-driven tool-use trajectories
Based on this data, we evaluate the state-of-the-an models, compare representative tool-selection strategies, and assess agentic tool-use settings beyond individual LLMs. For (3) the evaluation metrics, we report trajectory-aware metrics alongside final-answer accuracy: Trajectory Exact- Match and Trajectory Inclusion (whether the required tools are invoked and the correct order), Tool-Usage (schema constraints, formats, and value checks of the tool's inputs), and an LLM-judge Trajectory-Satee Satisff score when gold traces are unavailable. Together, these evaluations treat tool- primary skill, with trajectory- aware metrics that explicitly measure trajectory and query complexity. Our key contributions are summarized as follows:
Dataset and tasks. We introduce high-quality, executable tool suites across various domains and tasks. We model trajectory complexities via different structures (parallel and sequential) and various tool counts. Queries aligned with trajectories with different difficulty levels.
Evaluation suite. We evaluate state-of-the-art LLMs and representative tool selection strate- gies. We also include agentic tool-use in the evaluation suite. Beyond final-task accuracy, we introduce trajectory- metries better capture tool-use capability.
Insights and guidance. We provide analyses based on our evaluation, and reveal important insights, such as models' struggling to infer correct tools from indirect queries, the bottleneck of the transition from short to mid-length trajectories, limitations of retrieval based selection. These insights inspire future development of a more precise and reliable tool-use mechanism.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
LLM Agents. LLM-based agents treat large language models as the "brain" that plans, acts, and reflects while interfacing with external tools and environments (Wang al. 2024] Guo al., 2024). Early formulations follow think-ac act loop: ReAct interleaves chain-of-thougle with grounded ac-
tions (Yao et al. 2023b; Wei et al. 2022), and Reflexion adds self-critique and memory to correct future behavior (Shinn et al. 2023). Building on this, tool-augmented frameworks formalize func- tion calling, tool selection, and state management to operate retrieval, code execution, and other systems (Schick et al. 2023 Lewis et al., (2020). These designs have made LLM agents effective and flexible across domains He et al., 2024b] Trivedi et al., 2022; Li ct al. 2023b). Web agents
perceive pages and perform UT-level actions for tasks like shopping and booking (Yao et al. 2022
Zhou et al. 2023 He et al., [024a]] iZheng et al.
2024); code agents iteratively edit, test, and debug
multi-file projects (Yang al., 2024 Honn et al. 2024 embodied agents couple language with per-
ception and control (Kim et al.
et al. [023] Zhang et al. (024a); and scientific agents
use tools and simulation for literature review hypothesis generation, and experiment planning (Park
ct al.
2023
Ren et al.
2025).
Tool-use in LLM agents. Tool use is a core capability of LLM agents and has drawn substantial interest (Qu ct al. 2025 Qin et al. 2024). Methods advance when and how models invoke functions:
RL approaches (ReToool TooIRI, OTC) optimize selection and calling (Feng et al.. 2025 Qian al,,
2025a Wang et al.) (025); supervised/instrutiin tuning (e.g., Toolformer; small -odeel instruction
tuning) teaches adherence to schemas (Schick et al. 2023) [hen et al.) 2024);; and feedback frame- works (TRICE) add self-correction (Qiao et al.) 2023). Complementary evaluations measure these abilities: Gorilla grounds calls in public APIs (Patil et al. 2024); the Berkeley
Leaderboard scores cross-domain exccution (Patil et a ToolBench scales multi-step RapidAPI tools (Qin et af. 2023); TooIQA targets tool-augmented reasoning (Zhuang et al.) 2023); ;; and Meta- Tool probes tool awareness and selection (Huang et al., 2023). Together, this work advances both capability and measurement for tool-using agents. However, existing benchmarks seldom treat tool use as the primary objective. Most prioritize end-task scores and overlook trajectory-level signals, limiting the insight into agent behavior and opportunities for improvement. Therefore, a benchmark with a figg-fideitty, diverse tool set and realistic, task-driven queries spanning difficulty levels, en- abling trajectory- awarr aware evaluation and diagnosis, is still urgently needed.
3
DATA CONSTRUCTION
To construct the data, we first select the candidate tools to be included in the benchmark (Section and then curate ground truth tool-using trajectories and corresponding queries (Section 3.2
3.1
3.1 TOOL SET CURATION
Following (Qin et al. 2023), we source practical tools from RapidAPI, which contains many APIs for real-world tasks. To keep the effort tractable, we curate data for ten representative domains: travel, mapping, finance, weather, e-commerce, news/media, gaming, email, education, and music, where LLM agents are especially popular and usefu
Although thousands of APIs are available from RapidAPI, many of them contain unclear descrip- tions or parameters, and some are even non-executable. Therefore, we carefully select and revise the APIs based on the following four requirements.
(1) Executable tools with meaningful outputs. We validate each tool by executing it across param- eter combinations and discard those that error out (details in the Appendix [...] For the remainder, we use an LLM to summarize outputs and formats, removing tools whose outputs are semantically trivial, that is. do not contribute useful information.
tool descriptions. Because many APIs ship with sparse or vague doc- umentation, we refine API descriptions by combining the original description with empirical I/O observed during validation. For instance, the original description is just 'Get price (symbol)," yet empirical calls reveal it enforces a 50-item mage_size cap and returns {price, currency, timestamp); we merge these behaviors into the clarified description.
(2) Clear,
(3) Minimal functional overlap. We deduplicate identical tools/APIs (e.g., multiple feight-search endpoints) to avoid ambiguity and ensure deterministic trajectory evaluation, keeping a single repre-
Mooe domains can be added following our data-generation pipeline
3

---

Published as a conference paper at ICLR 2026
Parallel
Sequential
Task lype
Tool Graph Construction
Task Complete Travel Arrangemnt
Source Tool: Get All inta ampurt codes
lask description: This involves booking complete trip for
Tool Oulput:
of IATA airport codees
Simple query
Connected tools: [
Tool Trajectory
Help plan
aupoortintor get info airport using the IATA code airporedttar, get weather of an airport using (ATA code ....
Toal I: Search horess
Chicago lo Zurich from
Required_ Requireed_paroneten (
10/1/2025 to 10/0/2025. 5, First
"city": 'Zurich', ,
search flights then find proper
"checkin_datee date'; '2825-18-01',
people need two rooms. Find hotels with good reviews.
Task sequence
Query
'checkout_ date'; 18-55-1--8")
Name: Geographic Search
am planning trip among the top 10
Optional_paree paranetrrrt
Hotel Search Hotel
prefer northern and show me reciews of the available hotel the
'sort order"; "REVIEN",
Hard query
Details
'roon': 2}
Tool Search Alylis
'people': 4,
Zurich would
My family are planning a short
fall escape from Chicago and
good choice.
hmad city list and
progrssively
Task: this chain starts with a
down
Tool Trajectory
most northem city.
Required aequiree__parreteerr p
We aim to stay there for the first
to single hotel, each
Tool Download citees
"depantutre"; "Chicago", 'arrival': 'Zurich')
week of October. My wile and
sepp prwirling a norrecary
Required parameters: ("Eii::"""""'' 'limitt:
10)
have two children but
ID for the nest.
Tnol 2: Search hotel locations
would to enjoy sweet
Tool sequence: Download
Requiined paraneters:
couple nights. Please find us
chiin Sharch
('hotel id": '788008849")
someswhere with good record.
Aocations - Hore details
An illustration of data in TRAJECT- Bench. The left side demonstrates the parallel query where tool trajectories are created based on real task types and then queries with two difficulty level are generated. The right side shows the generation process of sequential queries, where a tool graph first built, then task sequences are manually designed and finally detailed queries and trajectories are created.
Figure I:
sentative per function. Closely related tools are kept when they introduce distinct parameterizations to increase task complexity. This is done with the help of LLMs, followed by manual verification.
(4) Controlled tool complexity. We manually keep tools with strong parameter complexity (number of fields, types, and constraints) and remove some with simpler parameters (e.g., inputs needed), ensuring the set contains rich tools to stress-test tool-use competence. Finally, we obtain a high- fidelity tool set T. Details and examples of the curated tool can be found in the Appendix A.
3.2 TASK-DRIVEN QUERY GENERATION
To evaluate LLMs' tool use in realistic settings while keeping the evaluation controllable, we syn- thesize queries from real-world task types, which denote categories/families of tasks that users can adopt the agent for, such as booking flights and arranging stays in the destination (without details of dates or locations). Specifically, as shown in the left side of Figure for each domain we manually collect representative task types (e.g.. "real-time trip monitoring & assistance" for travel; "creat- ing math & science study materials" for education), and provide a precise description for each task type. For every task type, we create tool-use trajectories with details and pair them with aligned user queries. By templating from task types, we minimize human efforts in collecting and annotating data, yet keep queries faithful to real use cases. This also enables diverse, controllable, and scalable trajectrajectory data across domains.
We consider two basic trajectory structures: (i) parallel, where tools operate independently (the choice/execution of one tool does not depend on others); and (ii) sequential, where tools form a chain and later steps depend earlier o Details of trajectory and query construction are provided as follows.
3.2.1 PARALLEL QUERY
Trajectory generation. For each domain, we prompt an LLM to synthesize valid, logical tool- using trajectories from a task-type description and the domain's 's available tools. To keep evaluation scalable and comparable, we enforce two rules: (i) the plan must use specified number of tools (typically 3-10+); and (ii) tools are presented as parallel-ready each call is self-contained (inputs fixed up front) and independent of the others. Parallel trajectories encoded as unordered sets of tool calls with fully specified inputs. For example, Figure shows that combining hotel and flight search APIs enables end-to-end travel planning.
Query generation. After trajectories are finalized, we pair each trajectory with user queries de- signed at two difficulty levels:
We focus parallel and chain structures to balance data cost and evaluation reliability. We present exper- iments hybrid structures in Appendix C and leave richer graph topologies for future work.
4

---

Published as a conference paper at ICLR 2026
Simple version. Provide straightforward and precise instruction that explicitly specifies the need for tools and their key parameters, such as the example in Figure where the simple query gives direct and detailed requests for travel plan.
Hard version. Present more challenging. indirect request that conveys the same con- straints via natural cues and implications rather than explicit instructions. This mirrors real interactions where users state goals colloquially (e.g.. "hotels with good record" rather than "sort hotels by review score" as shown in Figure D
After the generation, both trajectories and queries go over L.LM-based automatic validation and human inspection to ensure the quality and reduce ambiguity. By jointly varying (i) the number of tools per trajectory and (ii) the difficulty level of paired queries, we obtain a comprehensive yet well-controlled evaluation of LLMs tool-use capabilities.
3.2.2 SEQUENTIAL QUERY
Sequential queries require strong dependencies between tools within the trajectory, which makes difficult for an LLM to directly propose a correct chain end-to-end. To address this, we first build a directed tool graph GT = (V, 5) that captures how tools can feed one another: each tool in V is node, and we add directed edge i + l2 to the edge set E, when the information from the output of t1 can be used as input parameters of tz. For instance, as shown in the right side of Figure I the IATA codes returned from the GetAllLATA API can be used as the input for airportInfo API to retrieve detailed information of a specific airport, so we connect them.
Given Gr. we generate representative tasks and corresponding tool sequences with controlled number of tools (e.g.. sequences for each number of tools). Unlike the parallel setting, these sequences manually created as trajectory templates ensure logical coherence, e.g., & - tz - ... In.raj. where naja denotes the number of tools involved. Each template explicitly specifies the parameter bindings between adjacent tools, i.e., which output results from 4 populate which input fields of f++1. For instance, the task sequence in Figure [ describes a general 'city-jhotel location-ghotel detail' chain without details such as city name or hotel id.
Similar to the generation of parallel trajectories, we then prompt an LLM to create trajectories with details and corresponding user queries from each template (5 per template), yielding straj = {ti (params)) - ... - Enewy (params)} together with seg as shown in Figure All trajectories and their paired queries undergo automated validation and human review to ensure validity, task alignment, and overall quality. More details of the generation process and datasets are shown in the Appendix A.2 This pipeline produces coherent tool chains and tightly aligned queries varied depths (number of tools involved), enabling scalable and transparent evaluation.
Finally, the whole dataset contains 1228 tools and 5670 queries (details in Appendix A.2 A.22
4 EVALUATIONS ON TRAJECT-BENG
We comprehensively evaluate LLMs' agentic tool-use on TRAJECT-Bench, focusing on key re- search questions that highlight its benefits and potential:
RQL. When and why do LLMs succeed or fail at tool-use?
RQ2. Can retrieval-augmmentdd selection improve tool selection and parametrization?
RQ3. Do agentic methods, including training and inference, improve tool-use capability?
4.1 EVALUATION SETTINGS
Test (query) methods. When evaluating LLM's own capability in calling proper tools to solve queries, we mainly adopt the direct query where user queries and tools are provided the model. We provide additional results using Chain-of Thought in Appendix C
Tool selection strategy. Unlike existing benchmarks (Qin et al. 2023 Huang et al. 2023) ship only retrieved tools (not the full tool set) during evaluation, we treat the tool selection strategy crucial part of tool-using, especially when the tool set is large and exceeds the model's 's context window. We evaluate three common tool selection strategies (labels shown in Table44. all- aill-povvie the full tool set in context; domain provide only tools from the query's 's domain; and retrieval retrrevee a subset based on the query and tool descriptions. For retrieval, test two widely used embedding models,
5

---

Published as a conference paper at ICLR 2026
M---M-n (Solatorio, 2024) and bge-large-en-v1.55 (Xiao al. 2023), plus tool- retriever, ToolBench-IR (Qin et al. 2023). Unless noted, we retrieve 20 tools by default.
Test models. We test on wide range of state-of-the-art LLMs: Claude family, including Claude- 3.7 cant 2024) and Claude-4 (ant 2025); Gemini family, including Gemini-2.5-pri and Gemini-2.5- flash; GPT family, including m4-mini and gpt-oss- 120B; Qwen3-235b-A225 the latest generation of LLMs in the Qwen series; DeepSeek-V3. V3.1, the latest DeepSeek model; and Kimi-k2, the latest
MoE model from MoonShot AI.
Agentic evaluation. Despite the tool-using capability of models themselves, one highlight of our benchmark is the evaluation of models' agentic tool-using capability. We consider agentic capabilities D 'The first one is the capability specifically trained in the models. The SOTA LLMs, such Claude, Gemini, Deepseek, and Kimi-k2, all include tool-using in the training and directly support tool-calling, e.g., the "tools" input in cl client.. messages create of Claude APIs, and do not need to additionally list them the context of the prompt. Therefore, we provide evaluations on these models' internal tool-using capability and compare with the results of providing tools as context. More details can be found in Appendix B
We also evaluate LLM agents for tool-use. Given diverse designs, we focus on ReAct (Yao et al., 2023b) foundational blueprint for many real agents (e.g., MetaGPT (Hong et a (0224). We pair ReAct with over multiple turns, we test two retrieval modes: static (retrieve once from the query) and dynamic (retrieve at each reasoning/execution turn).
and retrieval-based tool selection. Because ReAct reasons and executes
Metrics. Unlike existing benchmarks, which mainly focus on the final performance, we con- sider two categories of metries: trajectory-awart metries and final performance metrics. For the trajectory-aware metrics, we consider (1) Exact match (EM), which compares the predicted tool- using trajectory and the ground-truth ones to check if the predicted tools (names, not parameters) are exactly the same as the ground truth ones; (2) Inclusion, which measures what proportion of ground truth tools is included in the predicted tool trajectory; (3) Tool Usage (Usage), which checks if the predicted tool parameters match the ground truth ones; and (4) trajectory satisfaction (Traj- satisfy), where we prompt an LLM judge (Claude-4 by default F to determine to what extent predicted trajectory can solve the user query, and by using this metric we mimic a real- real-world sce- nario when the ground truth is not available. For the final performance metric, we report (5) Acc, which measures if the predicted final answer matches the ground truth answer by prompting an LLM judge. For the retrieval- based methods, we include the (6) retrieval rate, which measures what proportion of ground truth tools are retrieved. More details can be found in Appendix B
4.2 RQI. WHEN AND WHY DO LLMs SUCCEED OR FAIL AT TOOL-USE?
We first evaluate the tool-use capability of individual LLMs. For each evaluating query, we feed the LLM both the query and the available tools as the context. Since the full tool set is beyond the length of the model's context window, we focus on tools related to the same domain of the query, e.g., we only provide Travel tools for query in the Travel domain. We report metries mentioned in Section in Table 2 for parallel data and Table 3 for sequential data.
Overall tool-use performance analysis. Generally, we observe that Gemini-2.5-pp performs best on simple queries. while Claude-4 ties with it on the hard versions. Less capable models, such as Claude-3.7 and Gemini-2.5 2.5 lag behind. The LLM-judge Traj-Satisfy score also tracks EM closely (e.g., Claude-4: 8.549 +-> 0.846 on simple; 4.882 <>> 0.445 on hard), indicating the judge is an effective proxy for EM.
In addition, we make the following key observations. (a) Most models perform well on the simple versions but struggle on the hard versions. For most metrics on most models, we observe clear gap between simple and hard; for example, EM for Claude-4 drops from 0.846 + 0.445 and for Gemini-2.5-pro from 0.851 + 0.442 under direct prompting. This suggests that when tool choices and constraints must be inferred from indirect cues, models often miss the exact tool set and pa- rameters, and future efforts should focus more on these more complex cases. (b) For nearly every
WWe note that agent structures are very diverse, and we include additional experiments in Appendix C We will keep exploring other agents and extending the TRAJ Bench..
"This evaluation does not require advanced models, and we also leverage smaller and open-source models in Appendix C

---

Published as a conference paper at ICLR 2026
model, parallel-siuple obtains better performance than sequential. For example, the EM, Inclusion, and Usage of Gemini-2.5-pro (0.851, 0.854, 0.835) on simple parallel queries are higher than those on sequential queries (0.807, 0.821, 0.809). suggesting that inter-step dependencies and ordering in- troduce additional challenges for both tool selection and parameter determination. This motivates future tool-use training and inference methods that explicitly model dependency and order. (c) In- clusion typically exceeds EM, especially on hard parallel and for weaker models. For most hard cases, there is a pronounced Inclusion-EM gap-e.g.. 0.135 vs. 0.554 for Claude-3.7 and 0.216 vs. 0.538 for Gemini-2.5 1omini-2.55-iaen 5--asshh------- indicatini that models fail to recover the complete set.
Tool-use scaling analysis. Since the user queries can be very complex in practice and can involve many tools, we analyze the model's tool-use capability when the number of tools in the (ground truth) trajectory increases. As an illustration, we present the results on both simple and hard queries on models from different families in Figure 2 where the x-axis is the number of tools and the y- axis is the metric EM. As the number of tools in a trajectory grows, all models experience clear decline in performance, with the steepest drop occurring between three and five tools. Among the evaluated models, Claude-4 and DeepSeek show the strongest generalization, maintaining relatively stable performance even at longer tool chains, whereas 04-mini and Kimi-k2 collapse sharply be- yond seven tools. These results reveal that scaling tool use is a universal challenge, where the main bottleneck lies in the transition from short to mid-length trajectories. and underline the importance of improving models' long-horizon tool-use and error recovery capabilities.
Deeper analysis on tool-use failures. We further take look at failed cases to identify common failure patterns in tool use, and detailed examples are presented in Appendix [D] (a) Similar tool confusion. This refers to the cases when models face tools that are partially overlapping capabili- ties but distinct scopes, inputs/outputs, or constraints (e.g., confusing between Spotify: Search vs. YouTube Music: Search). Distinguishing among these tools requires more delicate understanding of the tools' exact functionalities, parameters, outputs, etc. Existing models need more improvement to obtain a precise tool match facing such distractions. (b) Parameter- blind tool selection & us- age. In some cases, models overlook tool parameters (values, formats) when selecting tools, relying primarily on tool descriptions. This can lead to selecting tools with wrong parameters, cascadiin failures in downstream steps, and degraded end-task accuracy. We also notice that this failure is much rarer for Kimi-k2, suggesting that delicate agentic tool training can effectively improve this problem. (c) Redundant tool calling. This refers to cases where the model invokes more tools than necessary. We observe two forms of redundancy: (i) related-bus sordd-bu-unn nutt-mhelpful calls- e.g.. for a query about airport information in Zurich, the model additionally calls "Get All IATA airport codes," which returns every code irrespective of the city name provided in the query: and (ii) unrelated calls-e.g.. invoking railway tools when the task asks only for flight information. The first pattern likely reflects a conservative ccover-all-base or weak intent grounding. Since redundant calls can cause inflated latency and cost and even errors, precise tool-calling trajectory data should be included in training for improvement, and careful vali- dation can also be adopted.
strategy, whereas the second is more consistent with hallucination
(d) Struggle to infer intents from hard queries. When facing indirect queries, models often misinterpret the user's intent, leading to entirely inrelevant tool selections.
Together, these failure patterns show that models struggle with accurate tool choice and parameter use in complex scenarios, underscoring the need for better training and inference strategies.
dbude-4 4
0.60
cloude
gemini 2.5
gemini
mini
0.5
0.42
DeepSeek
DeepSeek
0.50
0.55
0.80
Kim-kz
0.45
Kimi-22
0.75
0.40
0.35
0.43
30
Tool Number (tool num)
Tool Number itool num)
Figure 2: Figures illustrating model's scaling tool-use behavior. The x-axis denote the number of tools in the trajectory and y-axis denotes the metric EM. Left is for simple queries, while right for hard queries.
7

---

Published as a conference paper at ICLR 2026
Table Results individual model's tool-un use capability with domain domain-specific tools on parallel queries.
Simple
Hard
Model
EM
Inclusion
Usage
Traj-Satisfy
Acc
EM
Inclusion
Usage
Traj-Satisty
Acc
Claude-3.7
0.676
0.746
0.706
6.831
0.714
0.135
0.554
0.603
1.582
0.246
Claude-4
0.846
0.855
0.839
8.549
0.905
0.445
0.668
0.794
4.882
0.517
Gemini-2.5-llash
0.714
0.759
0.784
7.215
0.782
0.216
0.538
0.666
2.340
0.263
Geminl-2.5-pro
0.851
0.854
0.835
8.599
0,911
0.442
0.652
0,785
4,849
0.498
GPTS-mini
0.817
0.825
0.819
8.236
0.834
0.155
0.589
0.626
1.701
0,380
o4-mini
0.823
0.830
0.824
8.316
0.863
0.376
0.629
0.726
3,849
0.472
gpt-oss:120b
0.652
0.667
0.715
6.588
0.726
0.174
0.559
0.671
1.909
0,387
DeepSeck
0,833
0,836
0.829
8.417
0,889
0.439
0.669
0.757
4.817
0.458
qwenn3-2350-A22B
0.844
0.856
0,809
8.529
0.898
0.440
0.667
0.796
4.828
0.479
Kimi-k2
0.815
0.876
0.872
8.236
0,902
0.321
0.666
0.772
3.522
0.448
Table 3: Results of individual model's tool-use capability with domain-speccficc tools on sequential queries.
Model
Inclusion
Usage
Traj-Satisfy
Ace
Claude-3.7
0.583
0.724
0.584
6.010
0.573
Claude-4
0.819
0.832
0.775
8.243
0.813
Gemini-2.5-flash
0.613
0.695
0.714
6,219
0.652
Gemini-2.5-pra
0,807
0.821
0.809
8.119
0.848
GPT5-mini
0.693
0.715
0.692
7,042
0.677
04-mini
0.789
0.807
0.748
8.134
0.761
gpt-oss:120b
0.538
0.619
0.694
5.546
0.653
DeepSeek
0.825
0.849
0.811
8.305
0.823
qwen3-235b-A228
0.824
0.837
0.772
8.194
0.791
Kimi-k2
0.821
0.827
0.793
8.363
0.833
4.3 RQ2. CAN PARAMETRIZATION?
SELECTION IMPROVE TOOL SELECTION AND
Since tool sets can be large and diverse in practice, it preferable to first narrow them to a smaller, relevant subset avoid adding irrelevant tools the context. Retrieval-basee strategy is pop- ular way to achieve this, and we evaluate different embedding models and different retrieval pools experiments find
what extent retrieval can help. The results in Table suggest two clear findings. (a) When the retrieval pool is already restricted to domain-relater tools, retrieval adds little benefit for simple queries. For both Claude-3.7 and Claude-4, EM and accuracy remain nearly the same across embedding models, with ToolBench-IB showing only marginal improve- ments. (b) Retrieval becomes a severe bottleneck for hard queries. We notice that the retrieval rate significantly drops (merely over 50% for most models) and all the performance metrics drop sharply compared with the non-retrieval cases. The core issue may that retrievers heavily rely on semantic similaritics and fail to capture underlying intents and steps from implicit queries. There- fore, they struggle to correctly identify necessary tools, causing cascading failures in tool selection and parameterization. Together, these findings reveal limitations of retrieval-based tool selection methods and call for a better strategy.
Table 4: Results of individual models combined with retrieval-based tool-selection strategy.
Clande-3.7
Simple
Hard
model
Retriecal paol
Retrieval
EM
Inclusien
Usoge
Traj-Satidy
Retrieval rale
Usage
Traj-Satidfy
bir-borge
0.665
MinMMiM
Domain
0.913
0.685
0.991
0.749
7,57
0.717
0.099
04001
0163
0.265
0.100
All
0.568
0.645
0.751
0.745
6.113
0.600
0.460
0.02
0.003
0.641
0.140
0,02
TooILMII
Dumain
0.945
0,703
0.814
0.778
7.142
0.755
0.578
0.030
0.19
0.688
0.294
0.130
All
0377
0.622
0.681
0.788
6.700
0.698
0.475
0124
0.425
0.656
0.299
0152
Chunded
Simple
Hard
bgr-borge
0.986
0.664
0.785
0.876
0.012
0.656
0.111
0.189
Mll-MiniLMM
0.81%
0.355
0.77
0.832
To-IIAI-IR
Domain
0.945
0.906
0.928
0.815
9.117
0.916
0.578
0.078
0.420
0.60
0.385
0.341
All
0.877
0.852
0.961
0.764
8.613
0.879
0.475
0.014
0.308
0653
0.190
0.164
8

---

Published as a conference paper at ICLR 2026
Table 5: Evaluation of model's inherent agentic tool-use capability, denoted as "agentic". "context" denotes the results providing tools as context, same with T
Simple
Hard
model
EM
Inclusion
Usage
Traj-Satisty
Acc
EM
Inclusion
Usage
Traj-Satisty
Acc
Claude-4
Agentic Coutext
0.832
0,868
0.816
8.407
0.893
0.440
0.637
0.751
4.828
0,486
0.846
0.855
0.839
8.549
0.905
0.445
0.668
0,794
4.882
0.517
Gemini-2.5-pro
Agentic
0.828
0.866
0.876
8.367
0.917
0.416
0.674
0.768
4,564
0,503
Context
0.851
0.854
0.835
8.599
0.911
0.442
0.652
0.785
4.849
0.498
Deepseek
Agentic
0.819
0.857
0.756
8.256
0.845
0.416
0.682
0.783
4.564
0.481
Context
0.833
0.836
0.729
8.417
0.889
0.439
0.669
0.757
4.817
0,458
Kimi-k2
Agentic
0.853
0.977
0,893
8.620
0.951
0.315
0.717
0.765
3.456
0.437
Coutext
0,876
0,872
8.236
0.321
0.666
3.522
0,448
4.4 RQ3. Do AGENTIC METHODS, INCLUDING TRAINING AND INFERENCE, IMPROVE TOOL-USE CAPABILITY?
Despite the individual model's capability of using tool information as the context, agentic methods also developed in existing literature, including two major categories: one is the agentic tool-use training undergone by most SOTA models (Team et al. 2025 Yang et al. 2025). and the other is design LLM agents to adopt tools (Yao et al. 2023b). For the first category, we evaluate four representative models that inherently support tool-use, and show results in Table 5 where we also provide results of context- based evaluation as a baseline. Table 5 shows that the agentic tool-use capability is similar to the context- based baseline for most models on both simple and hard queries.
For the second category, we evaluate Claude models on both parallel and sequential queries by com- bining retrieval with ReAct. We consider two settings: static mode, where a subset of tools is retrieved once based on the user query, and a dynamic mode, where retrieval occurs before each thought and action. TooILM-IR is used the retrieval backbone in all experiments. The results in Tables and 7 show that ReAct consistently improves tool-use performance compared with individ- ual models in Table 2 For example, on parallel hard queries, Claude-4 improves from 0.445 EM (Table 2} to 0.463 EM with ReAct ()). and the performance further boosted via dynamic retrieval to 0.473 EM. Similarly, Claude-3.7 gains from 0.135 EM 0.186 EM with domain tools and 0.296 EM under dynamic ReAct. These show that iterative calling tools based on execution results offer stronger basis for accurate tool retrieval and usage.
Overall, these findings suggest that agentic tool learning and inference serve as an effective and robust way improve a model's tool-use capability.
Table 6: ReAct results on parallel queries combined with different retrieval strategy.
Simple
Hard
Model
retrieval mode
FM
Inclusion
Usnge
Traj-Satisyy
Acc
EM
Inclusion
Usage
Traj-Satisfy
Acc
Domain tool
0.735
0.816
0.752
7,630
0.612
0.641
1,930
0.307
clande-3.7
Static
0.762
0.841
0.791
7,840
0.803
0.093
0.432
0.697
1.180
0.148
Dynamic
0.814
0.876
0.805
8.220
0.831
0.296
0.581
0.703
3,080
0.346
Domain tool
0.892
0.915
0.864
9.160
0.928
0.463
0.670
0.797
4.804
0.418
claude-4
Static
0.886
0.912
0,872
9.116
0.916
0.031
0.424
0.668
0.403
0.139
Dynamic
0.933
0.951
0.875
9.422
0.948
0.472
0.703
0.736
4.811
0.438
Table 7: ReAct results on sequential queries combined with different retrieval strategy.
Sequential
Retrieval Mode
EM
Inclusion
Usage
Traj-Satisfy
Acc
Domain tool
0.615
0.735
0.625
6.384
0.634
claude-3.7
Static
0.609
0.692
0.644
6.266
0.603
Dynamic
0.651
0,749
0.651
6.574
0.634
Domain tool
0.827
0.846
0.829
8.492
0.842
claude-4
Static
0.817
0.833
0.825
8.059
0.811
Dynamic
0.849
0.881
0.833
8.573
0.917
5 CONCLUSION
This work introduces a comprehensive tool-use benchmark that focuses on the tool-use trajectory. By constructing large and diverse executable tool suite, modeling trajectories of varying structures

---

Published as a conference paper at ICLR 2026
and scales, and pairing them with queries of different difficulty levels, provide realistic and rigorous setting for evaluation. Our trajectory-aware metrics go beyond final-answer accuracy, enabling clearer understanding of where and why LLMs succeed fail in tool use. Through comprehensive evaluation and analysis, our henchmark not only highlights the current strengths and limitations of state-of-the-art models but also offers actionable insights for improving agentic tool-use capabilities. We hope this benchmark establishes a foundation for systematic progress in developing LLMs that can reliably plan, select, and execute tools in complex, real-world scenar- ios. We will also keep including more diverse domains, agent structures, and extend to dynamic environments to maintain PRAJECT-Benchine the long run.
ACKNOWLEDGMENT
Pengfei He and Jiliang Tang are supported by the National Science Foundation (NSF) under grant numbers CNS2321416, IIS221 2032, IIS2212144, IIS 2504089, DUE2234015, CNS2246050, and DRL22405883, the Michigan Department of Agrieulture and Rural Development, US Dept of Com- merce, Gates Foundation, Amazon Faculty Award, Meta, NVIDIA, Microsoft and SNAP. Yue Xing supported by NSF DMS 2515194, Open Philanthropy, NVIDIA Academic Grant Program and Google Cloud Research Credit.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We acknowledge the ICLR Code of Ethics and ensure that no concerns regarding the Code of Ethics arise from our work. Our study does not involve human subjects, personal sensitive data, or experiments that could cause harm. All data used are either synthetic or publicly available under appropriate licenses, and we have adhered to principles of fairness, transparency, and reproducibility throughout.
REPRODUCIBILITY STATEMENT
We
and codes in TaolData- pataa-ppllic-25550 mentioned in Appendix B
provide data
science/// We provide additional details in
section 4.1
REFERENCES
The claude 3 model family: Opus, sonnet, haiku - model card. Technical report, Anthropic, 2024.
URL httph:tt///t/ anthropic com/ claude modell card Accessed 2025-09-20.
System
card:
Claude
opus
& claude
sonnet
Technical re-
port, Anthropic, May 2025.
URL https: hts:://wwwr /www. anthropiicicc.... com/
pddf Accessed 2025-09-20.
Aili Chen, Xuyang Ge, Ziquan Fu, Yanghua Xiao, and Jiangjie Chen. Travelagent: An ai assistant
for personalized travel planning. arXiv preprint arXiv:2409, 08069, 2024.
Zhendong Chu, Shen Wang, Jian Xie, Tinghui Zhu, Yibo Yan, Jinheng Ye, Aoxiao Zhong, Xuming Hu, Jing Liang, Philip $ Yu, et al. Llm agents for education: Advances and applications. arXiv 2025.
preprint arXiv:2503.
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in Ilms. arXiv preprint arXiv:2504. 1153,, 2025.
Daya Guo, Wenfeng Liang, Haowei Zhang, and et al. Deepseek-rl incentivizes reasoning in llms through reinforcement learning. Nature, 645:633 638, 2025. doi: 10. 00008844444880000-99094422 URL https::tt//wwt nature cwm htt.:/rr.....o.rttteesse//s////e 0111:/4411888002000-2-------
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: survey of progress and challenges. arXiv preprint 24X.:2402.00680 2024.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.139/9 2024a.
Pengfei He, Zitao Li, Yue Xing, Yaling Li, Jiliang Tang, and Bolin Ding. Make Ilms better zero-shot reasoners: Streeture-orientaated autonomous reasoning, arXiv preprint arXiv:2410.1900 2024b.
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta programming for multi-agent collaborative framework. International Conference Learning Representations, ICLR, 2024.
Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of Imm agents: A survey. arXiv preprint 40Xiv:2402.0227 2024.
Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu. Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, et al. Metatool benchmark for large language models: Deciding whether to tools and which use. arXiv preprint 1rXiv:2310.033 12888 128, 2023.

---

Published as a conference paper at ICLR 2026
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Serean Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training 1lms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025.
Callie Y Kim, Christine P Lee, and Bilge Mutlu. Understanding large-language model (Im)- powered human-robot interaction. In Proceedings of the 2024 ACM/IEEE international con- ference on human-robot interaction, pp. 371 371-380, 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmenter gener- ation for Knowledge-intensive nlp tasks. Advances in neural information processing systems, 33: 9459-9474, 2020.
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented Ilms. arXiv preprint Siti::2304002244 2023a.
Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, and Li- dong Bing. Shain-of-kkowledge Grounding large language models via dynamic knowledge adapting over heterogeneous sources. arXiv preprint arXiv:2305. 2023b.
Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. A survey on Ilm-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth, 1(1):9, 2024.
Jiageng Mao, Junjie Ye, Yuxi Qian, Marco Pavone, and Yue Wang. A language agent for autonomous driving, arXiv preprint 1rXiv:2311.100 10813, 2023.
Charles Packer, Vivian Fang, Shishir_G G Patil, Kevin Lin, Sarah Wooders, and Joseph Gonzalez. Memgpt: Towards Ilms as operating systems. 2023.
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceed- ings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST '23, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701320. doi: 0111445588113336667777
URL htt.htttts:ooooio.. org/600....00....
3606763
Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agen- tic evaluation of large language models. In Forty-second International Conference on Machine Learning.
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. Advances Neural Information Processing Systems, 37:126544 126565, 2024.
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Türs Gokhan Tur, and Heng Ji. Toolrl: Reward all tool learning needs. arXiv preprint urXiv:2504.1392 2025a.
Cheng Qian, Emre Can Acikgoz, Hongru Wang, Xiusi Chen, Avirup Sil, Dilek Hakkani-Türr Gokhan Tur, and Heng Ji. Smart: Self-aware agent for tool overuse mitigation. arXiv preprint 10Xiv:2502.11455.. 2025b.
Shuofci Qiao, Honghao Gui, Chengfei Lv, Qianghuai Jia, Huajun Chen, and Ningyu Zhang. Making language models better tool learners with execution feedback. arXiv preprint arXiv:2305. 13068, 2023.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolll: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16788 2023.
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng. Xuanhe Zhou, Yufei Huang, Chaojun Xiao, et al. Tool learning with foundation models. ACM Computing Surveys, 57(4): 2024.
12

---

Published as a conference paper at ICLR 2026
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji- Rong Wen. Tool learning with large language models: A survey. Frontiers of Computer Science, 19(8):188433, 2025.
Shuo Ren, Pu Jian, Zhenjiang Ren, Chunlin Leng, Can Xie, and Jiajun Zhang. Towards scientific intelligence: A survey of llm-based scientific agents. arXiv preprint arXiv:2503.24047, 2025.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of language model? arXiv preprint arXiv:2002.08910, 2020.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539- 68551, 2023.
Weizhou Shen, Chenliang Li, Hongzhan Chen, Ming Yan, Xiaojun Quan, Hehong Chen, Ji Zhang, and Fei Huang. Small Ilms weak tool learners: A multi-llm agent. arXiv preprint arXiv:2401.07324 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634-8655 2023.
Harmanpreet Singh, Nikhil Verma, Yixiao Wang, Manasa Bharadwaj. Homa Fashandi, Kevin Fer- reira, and Chul Lec. Personal large language model agents: A case study on tailored travel planning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 486-514, 2024.
Aivin V. Solatorio. Gistembed: Guided in-sa selection of training negatives for text embed- ding fine-tuning. arXiv preprint 1682222402.16829 2024. URL httptsss///aiii..rrttttrt abs/
2402. 16829
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. LIm-planner: Few-shot grounded planning for embodied agents with large language models. In Prooeedingg the the IEEE/CVF international conference computer vision, pp. 1998-3009, 2023.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534 2025.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving trieval with thfin-of-thought reasoning for knowledge-intensive multi-step questions. arXiv preprint arXiv:2212. 2022.
Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fu Fai Wong, and Heng Ji. Otc: Optimal tool calls via reinforcement learning. arXiv e-prints, pp. arXiv-2504, 2025.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 1866)) 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24831 2022.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint (0l::2505009888 2025.
13

---

Published as a conference paper at ICLR 2026
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agept-co interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528-50655 50-555005,,222 2024.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- cessing Systems, 35:20744 15:20744-20757, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances neural information processing systems, 36:1 1809-1182, 2023a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference Learning Representations (ICLR), 2023b.
Yang Zhang, Shixin Yang, Chenjia Bai, Fei Wu, Xiu Li, Zhen Wang, and Xuelong Li. Towards efficient Iim grounding for embodied multi-agent collaboration, arXiv preprint arXiv:2405 14314, 2024a.
Zheyuan Zhang, Daniel Zhang-Li, Jifan Yu, Linlu Gong, Jinchang Zhou, Zhanxin Hao, Jianx- iao Jiang, Jie Cao, Huiqin Liu, Zhiyuan Liu, et al. Simulating classroom education with 1lm- empowered agents. arXiv preprint arXiv:2406. 19226, 2024b.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is generalist web agent, if grounded. arXiv preprint arXiv:240100611 2024.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et Webarena: realistic web environment for build- ing autonomous agents. arXiv preprint arXiv:2307. 13854, 2023.
Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for I1m question answering with external tools. Advances in Neural Information Processing Systems, 36: 50117-50143, 2023.
14

---

Published as a conference paper at ICLR 2026
A
TRAJECT-BENCE DETAILS
We provide more details of the proposed IRAJECT-Bench.
A.1 TOOL SET DETAILS AND EXAMPLES
Overall statictics. We provide statistics of the tool set in TRAJECT-Bench in Table
Table 8: Statistics of tool set in TRAJECT-Bench
domain
tool num
Travel
131
Mapping
124
Finance
258
Weather
135
eoommecce
82
Mews_Media
Gaming
84
Email
70
Education
89
Music
137
Total
1228
Tool examples. We then provide some examples of tools in TRAJECT-Bench as follows.
"tool name"; "Airbnb listings: Prices and Availability by lat/Ing", "tool description": "Retrieves average prices, availability percentages and processed property counts within a 20, 11,0000-mess radius of geographical point, filterable by bedrooms and guest capacity. This analytics endpoint leverages the parent tool's access to daily- updated Airbnb data, providing insights into property availability and pricing trends to help users make informed decisions about accommodations.."
"required "pequireeepprreeeeeer"""" paraneters::
"name"; "year", "type": "NUMBER", "description": "the year", default": 2024 },
"name"; "lat", "type": "NUMBER", "description": "latitude",
default": 28.0852473 h
"name": "Ing", "ing", "type": "NUMBER", "description"; "longitude",
default": -16.7349700 h,
"namc"; "range", "type": "NUMBER", "description": "range in meters from latitude and longitude point", "default"; 500 ), "name"; "month", "type"; "STRING", "description"; "the month", default": "I"
},
[
"name": "bedrooms", "type": "NUMBER", "description": "number bedrocms", "default": 1, "name": "maxquestcapacity" "type"; "NUMBER", "description": maximur. guest capacity", "default"; 2
},
"API name"; "Prices and Availability by lat/ing", "domain name"; "Travel",
"outputiinfe info":
"The tecl provides statistical summary of Airbnb listings within apecified geographical radius. A successful response returns aggregated data, including the total number of properties, median prices for all and for available properties, and the overall availability percentage. This information is intended to help users understand local acconredation market trenda. The tool also returns simple error
15

---

Published as a conference paper at ICLR 2026
message object in cases of failure, such authentication subscription issue.",
"output_formact" "The output is a single, flat JSON object. The structure differs for successful and failed requests. A successful response contains keys like "totalProperties" medianPrice', and 'availability' contains a single 'nessage" key with a string value. There is no indication of nesting, arrays, or pagination."
response typically
"connected tools":
*tool name": "Airbnb listings: Listings by lat/Ing",
connect params":
"name": "lat", "description": "Can use the same latitude from the source tool's query" b,
"name": "Ing", "description": "Can use the same longitude from the source tool's query" b,
""name"; "range", "description"; "Can use the same or adjusted range based property density from source tool")
},
"tool name": "Priceline con Provider: Search hotels Locations by geolocation",
ccnnect params":
"name"; "latitude", "description": "Can use the same latitude from the source tool's query" }, "name"; "longitude", "description": "Can use the same longitude from the tool's query"
"tool name": "Billboard: Hot 100",
"tool description"; "Provides detailed information about Hil1board's Hot 100 chart, di splainng the current rankings of the most popular songs in the United States. This endpoint delivers access to Billboard's comprehensive music chart system, which includes not only the Hot 100 but also other major charts like Billboard 200, Billboard Giobal 200, and Artist 100. Billboard charts are the music industry's standard for measuring the popularity of artists, albums, and songs based on sales, radio airplay, streaming data, and other metrics, offering authoritative insights into current music trends and performance.", "required_parameteer
"name": "date",
"type": "DATE (XYYY-MM-DD)",
"description": "Set the date for which you want to get chart.", "default": "2020-03-18*
},
"opticnal haw.iiccal_ppammtteess.st [],
"API name": "Hot 100",
"domain name": "Music", "output_info"
"The tool returns a ranked 1ist of songs from the Billboard Hot 100 chart. For each song, it provides details such the artist, title, current rank, last week' s rank, peak position,
the number of weeks has been the chart. This data allowa users to track the current popularity and historical performance of top songs in the United States.", "output_formatt "The output is string-serialized JSON array of objects. Each object corresponds to song and its chart data.
16

---

Published as a conference paper at ICLR 2026
The 'artist' field contains ETML <<a>> tags linking Billboard's website. The weeks_on_chart field can an empty string, likely for new entries. Some string values may contain HTML character entitics (e.g., "
ccnnected tools":
"tcol name": "Spotify: Search", "connect params":
"name": "q", "description": "The song title and artist from Hot 100 can be used as search query"
"name": "type", "description"; "Can be set to 'tracks' to search for songs from the Hot 100 chart"
"tool name": "Shazam: search", "connect params":
"name": "term", "description": "The seng title and artist from Hot 100 can be used as search term"
}
"tool name"; "Youtube Music API (Detailed); Search", "connect params";
"name": "query", "description": "The song title and artist from Hot 100 be used as search query"
Execution errors of tools. As mentioned in Section 3.1
A.2 TRAJECTORY AND QUERY DETAILS
We provide more details of tool-use trajectories and queries in TRAJECT-Bench.
Trajectory statistics. The statistics of queries and trajectories in TRAJECT JeBJT-BBBecchh are shown in Table
Examples of task types for parallel queries. We provide some examples of task types used in parallel query generations.
"comain": "Travel",
"task name": "Find Proper Accommodations",
"task description"; "Find preper hotels based en some options. requires searching across different platforms, comparing options based on real-time data, and retrieving specific, granular details like policies, recent reviews, room availability that crucial for making booking decist decision.",
17

---

Published as a conference paper at ICLR 2026
Table Statistics queries and trajectories in TRAJECT-Bench
domain
simple parallel
hard parallel
sequential
Travel
200
200
170
Mapping
200
200
200
Finance
200
200
135
Weather
200
eCommerce
200
200
200
Mews Media
200
200
180
Gaming
200
200
185
Email
200
200
200
Education
200
200
200
Music
200
200
200
Total
2000
2000
1870
"tool classes": "Hotel", "Airbnb"
"task examples"; [
"query": "I'm planning a trip to London from Oct 10 to Oct 17 in 2025. I need to find the best hotel deal. Can you search both Booking. com and Hotels. com? want hotel with at Least 4-star rating and need see cancellation policy and some recent photos before decide.",
"tool list": [
"Best Booking, .oong..oonn com Hotel: find best booking.com
accomnodation"
"Hotels com Provider: Hotels Search",
"Booking.com stay:/propert.es/dddaalll/p
"Booking.com htap:/ppossrreee//ttat/////ootte
]
b
"domain": "Gaming",
"task name": "Tabletop & Strategy Game Assistance", "task description"; "This task focuses on the non-video game tools, previding user with rule-based assistance for traditional tabletop or card gane, such as calculating odds or determining the nathematically optimal movo.", "tool classes": "Card_Dice_Gar "Chess_Puzzless
},
"task examples":
"query": "I'm playing game of Blackjack and need some help. My hand is two 8s. According to basic strategy, what is the optimal move? Also, I think the deck is rich in high cards; the running count is +8 and there are about 4 decks left in the shoe. What is the True Count?", "tool list"; "BlackJack Basic strategy: Pairs", "BlackJack Basic Strategy: True Count"
The dealer showing
18

---

Published as a conference paper at ICLR 2026
"domain"; "Finance",
"task name"; "In-Depth Company Analysis",
"task description"; "This task involves creating holistic financial overview of specific company/stock by combining its reported fundamentals, current market data, and the consenaus opinion of professional analysts.",
"tool classes";
"Fundamental_Date Data",
Samalyst_Ratinge Estimates"
1,
"task examples":
"query": "Give me complete financial workup on Apple (AAPL)
I want and balance sheet. Also, pull the current institutional ownership percentage and any recent insider transactions. Finally, what is the consensus analyst price target for the stock?",
their latest quarterly income statement
"tool list":
"Financial Modeling Prep: Company Income Statement", "Financial Modeling Prep: Companies balance sheet statement",
"Mboum Finance: stce//inst.uttiooroweeeee..
"EOD Historical Data: Insider Transactions API", "YFINANCE Stock Market Data: Analyst Price Target"
Examples of task sequences for sequential queries. We also provide examples of task sequences used to generate exact sequential queries. is clear that we define coherence chain for each task and specify connections (parameter used for subsequent tools) among tools. Note that these sequences are general and do not contain any exact information such as detailed parameter values, they can serve as templates to create exact trajectories and queries fitting different realistic scenarios.
"comain": "Music",
"name": "Radio Genre Artist Profile Discovery",
"description": "This sequence starts by discovering radio stations specific country to identify music genre, then finds an artist in that genze on Spotify, and concludes by retrieving that artist's detailed profile.", "tool sequence";
"tool name": "50K Radio Staticns: Get Countries",
"use description": "Retrieves list of countries with radio stations to select country ID.", "param for next tool": "country.i 1,
"tool name": "50K Radio Staticns: Get Channels",
"use description"; "Finds radic channels selected country to identify a prominent genre.",
"aarm for next tool": "q"
"tool name"; "Spotify: Search",
"use description": "Searches for artists on spotify using the genre a query to get an artist's ID.", "param for next tool": "artist_id"
19

---

Published as a conference paper at ICLR 2026
"tool name": "Spotify _v: Artist Details (Single)", uuse description"; "Fetches the complete profile for the artist from Spotify using their ID.", "param for next tool":
1,
"tool number";
"domain": "Education",
"name": "DEL Resource Speaker Analysis Vocabulary Deep Dive Further Study", "description": "This sequence models student's workflow: finding DEI resource, identifying speaker, finding their TED talks, and then performing a deep vocabulary analysis on key term before exploring related learning resources.", "tool sequence";
"tool name": "DEI: Get.learriingeessuursss"""
"use description"; "A user gets a list of DEI learning resources and selects one, which provides 'title'"", "param for next tool": "title"
),
"tool name": "DEI: GetPeople",
"use description": "The 'title' is used to find an influential person related to that topic, providing their l "name"." ", "param for next tool": "name"
),
"tool name"; "TED Talks API: getTalks",
"use description"; "The persen's s 'name' is used the 'seekkrr' to find their TED talks, which returns 'talk "param for next tool": "talk_title" ),
"tool name": "Urban Dictionary: Define a Word Search",
"use description": "A key term from the 'talk title' used as the term' to look up its modern, colloquial, or slang meaning.", "param for next tool": "term"
"tool name"; "Dictionary: wordScarchTurkish*
luse description": "The same "term' is used as a 'query' to find its Turkish translation, providing 'anlam' (meaning)."
"param for next tool": "anlam"
"tool name": "Urban Dicticnary: Define a Word / Search",
uuse description": "The Turkish 'anlam" (meaning) is used as 'term' reverse-lookup to see its English slang definitions.", "para for next tool"; "topic"
b,
"tool name": "DEI: Getleammingeesarces"""
"use description"; "Finally, the original 'topic" from talk used to find other, related learning resources.", "param for next tool":
"tool number":
20

---

Published as a conference paper at ICLR 2026
Examples of queries and trajectories. We then provide examples of trajectories and queries in TRAJECT-Bench.
Parallel.
Example 1: Travel domain Tool Trajectory:
"tcol name": "Priceline con Provider: Hotel reviews", "tcol description": "Returns 1ist reviews for travel services such as hotels, cars, and flights. This endpoint part of the Priceline con API, which enables users to search and book travel accommodations and rentals. It leverages database of user-generated reviews to provide insights, helping travelers make informed decisions about their bookings.",
requiree parameters";
"name": "hotel id",
"value": "700022612"
"eptional parameters";
"name": "languages", "value": "en, fr"
"name"; "limit", "value"; 100
"name": vonly_veriffedd_uestt "value": true
eexecutee__uutpuue "["getHoteLReviews's ('error': l'status': Hotel.
Reviews: Retei.Reviews: number of reviews.', 'status_code's 11.822.8', 'time': 10.0715'333*
zeviews found as Offset value exceeds the
"tool name": "Booking.com v2:
cen_ languages",
"tool description": "Retrieves available language options for hotel information and booking interfaces. This endpoint leverages Booking.com com'' unofficial API access real-time data from hotels worldwide, allowing users to specify their preferred language when querying room availability, pricing, facilities, and policies as part of comprehensive travel planning.",
"required parameters": Il,
"optional parameters"; Il,
"executed eexecuted_outputt: "('data'; 11'rref'; ref'; 'Language: "code \ \ "en-
gb\") re_ref'; " 'Language: "c""de""""""n-u'"""
{' ref
:
r'_ref'; LLanguage: "oode
\">\"]\\">), "'_ref' 'Language: (o \
-ref'; "Manguge::"""oodd
""""" r'_ref'; 'Language 1' r''ref' "language:/"<oode":::
mx{")'), 1' ref': : 'Language: ""ooe"""""""""""""""""""""""""""
re_'
1' "'_ref'' ref': 'Language:
\">'"pt-pt\"">' ( ('_ref'; "
['_ref': "Languge:"""dde
: "[""ode\\::"EE\"">"" 1' re_'' (anguage::("ccool"=::: 'Language
r'_ref' :off:''Languun
{"}), re_ref': 'language: (codee>">> """"""" ('_ref':
Language: ( re_ref' "Langugge: (Vocde
21

---

Published as a conference paper at ICLR 2026
\"\'"a\""', ref': 'Language: ( "code\"; "rr:""ro'""> )
rref'; ( , r'-ref';
'Language
("""ooe\":\""h><"""""
ref'; 'Language; "cooel"
1² "r_reff;: """")",,""""ff"
r'_ref';
"onppase:""ooodeMM::M"\\""" 1'_ref';
ref': ( ("oogaaaee""codee:""""""
\": \">\"ru\"'),, -" ",""""""""""""""""""",,"" "'_ref': :
Lref:: "Marquaa::(""c"""" : =l\"code"":\aa"""1"1
'Language
-_ff"f" : "
I'__tt': ref': Lnogussei(1oods<:::111111111 ( "Language:
re
ret':
\": \"\\"o\")'}> ref' -"_ -ref'; "iagggge:
" r'_ref: : Language
:[""code\":\"ss""]1] r'_ref': [
\"I', 1' ref': re= :ag Lanpusee:""""""""::""""""""""" ("aggae
ref': ref': ( \">\"It\")",, re_ " -ref'; " (1 ('_ref': 'Language
{'
:[""code\":\"<"""" ['_ref': "Lnguue:(""coo>:
\")', (' ref'; ( (Cooee) "code"""\"=l""""] ref'; ("nguage
'message': 'Successful',
status' True?"
"tool name": "Cities Cost of Living: Get Cities List", "tool description": "Retrieves a comprehensive 1ist of all available cities in the database. This endpoint taps into a global dataset covering over 650 cities worldwide, providing access to the complete catalog of locations for which detailed living expense information is available. The parent service offers extensive cost-of-living data that helps users compare and understand financial requirements across diffcrent urban centers globally.",
"required parameters"; [],
"optional parameters"; [],
dereccuted_ouppure "('cities': I('ouutty' "Turkey', 'name' Kocaeli'), (' l'country'; 'Ceech Republic', 'name': 'Ostrava' ('country's fUnited States', 'name' 'Santa Clara'), country': Uniited States', 'name': 'oww City'), ('country': Unittdd Kingdon', 'name': 'Bournenouth" 1, l'country': Montenegro', 'name': 'Podgorica'), ,, ('country': 'United States 'name' 'Doledo'), l'country': 'United Kingdom', 'name' Milton Keynes';, ('country'; 'India', 'name' 'Mangalore'; (' country': 'Malaysia', 'name': 'Johor Bahru'), l'country': Canada', 'name'; \ \"st. John's John's)"), (' l'country': 'Pakistan',
name': 'Islanabad' 1, ( ('country': 'Norway', 'name' 'Troncheim ",, l'country': "Inii', 'name' 'Nagpur' ('country': Czech Republic', 'name': 'Olomouc'), ('country': 'Netherlands "name': " l'country': 'Ukzaine', 'name' Zaporizhzhya' ('country': 'India', 'name": 'Mysore'), ) (' country'; 'China', 'name' 'Beijing'; i, l'country': 'North Macedonia', 'name': "Skopje'), c'cuuutyy 'Philippines', name": 'Manila'), ( ('country': Netherlanda' 'name': Rotterdam'); ), !'country': 'raiwan (China)', 'name': "Taipei') l'country': 'Brazil', 'name': Golaani' l'country':
Taiwan (China)', 'name' 'New Taipei City' 1, ('country':
Austria', 'nane'; 'Salzburg'), }, i' ['country': 'Germany', 'name' 'Frankfurt'; ;, ( ['country': 'Turkey', 'name"; 'Diyarbakir'). (' l'country;; 'Canada', 'namo': 'North Vancouver') (' l'country': "India' 'nane': 'Ladhiana'), 1' l'country': 'Irral'' 'namc' "Jerssalmm) " c'cuutyy'' 'Afghanistan', 'name' 'Kabul'), country': 'Roaania', 'name': 'Iasi";, ('country': "United kingdom', 'nane" 'Bristol'), l'country': 'South Africa', name': : 'Port Elizabeth'), I, l'country': 'Ecuador', 'name' Guaysquil') "' l'countzy': 'France', 'name': 'Bordeaux') country': 'Georgia', 'name' B "Batumi',, l'country's 'Botswana
22

---

Published as a conference paper at ICLR 2026
'name' 'Gaborone') ,, ( l'country's 'Brazil', 'name' "Natal ", (' country': (Trinidad And Tobago', "name'; 'Port Spain ", (' country'; 'Belgium', nname'; 'Antwerp') ('countzy's Austria', 'name'; (Vienna'), l'country'; 'France', 'name':
1r "trajectory_typet" "parallel", "tool count": 3,
"# Travel Legistics Informationaaaaf 1. Hotel Reviews from Priceline com\n\nIhe a Friceline, Provider tool returned an error
attempting to retrieve hotel reviews. The specific error message was: "ootel.oee.eee as Offset value exceeds the number of reviews. reviews.\"nnnnne.... A'AnnnThis suggests that either the hotel ID provided (700022612) doesn't have any issue with the parameters used in the
No reviews found
reviews, or there request. To get valid hotel reviews, you would need to provide a different hotel ID or adjust the offset parametor. AnAnff 2.
Languages Supported by Booking.comln//nonn
con supports 45
languages across various regions. The complete 1ist includes: An English (UK): en-gbin- English (US): en-usin- German: deln- Dutch: nl French: frin- Spanish: es Spanish (Argentina): es
-arln- Spanish (Mexico): es-mx\n- Catalan: caln- Italian: itln-
Portuguese (PortugaL): pt-pt\n- - Fortuguese (Brazil): pt-brin-
Norwegian: noln- Finnish: filn- Swedish: svin- Danish: daln-
Czech: cs\n- Hungarian: buln- Romanian: zoln- Japanese: jaln-
Chinese (Simplified) zh-en Chinesc (Traditional): zh-twin-
Polish: plan- Greek: el Russian: ruAn- Turkish: trln-
Bulgarian; bg\n- Arabic: ar Korean: koln- Hebrew: heln-
Latvian: 1vln- Ukrainian: ukin Hindi: hiln- Indonesian: idln-
Malay: ms\n- Thai: thl- Estonian: st Croatian: brin-
Lithuanian: 1t\n- Slovak: skin Serbian: sr\n- Slovenian: siin- Vietnamese; viln- Sagalog: tile- Icclandic: is\n\n## 3. Cities for Cost-of-Living ComparisonslaNNThe Cities Cost of Living tool returned & partial 1ist of cities available for cost-of-living comparisons. The output was truncated, showing only the first 43 cities out of what appears to be a much larger database reportedly covering over 650 cities worldwide), \ninSome of the cities included the partial list:\n- Kocaeli, TurkeyIn- Ostrava, Czech Republicka- Santa Clara, United States\n- Iowa city, United States Bournemouth, United Kingdom\p- Beijing, Chinaln- Manila, Philippines\n- Frankfurt, Germany\n- Vienna, Austria\n\nPlease note that this is only small subset of the complete city list, as the tool output was cut off after cities. AnAnt CummarylatnI'we supported by Booking. com com (45 languages) and a partial list of cities available for cost-of-living comparisons. Unfortunately, the hotel reviews request returned error, available with the provided parameters. To complete your travel logistics preparation, you may need to request hotel reviews with different parameters and obtain the complete city list for cost- of-living compari sons." "o
gathered information on languages
no review data is
Simple query: I'm preparing travel logistics. Can you grab: Hotel reviews from Priceline. com; which languages Booking.com supports; and the city list for cost of living comparisons?
Eard query: I'm pulling together travel logistics for fall trip. Could you first scan recent guest feodback from major U.S. hotel aggregator and summarizo consistent pros/cons for central options? Next, check which languages the leading global booking portal actually supports across its site and app (not juat auto-translate)) and note any gaps. Finally, put together short list of major cities we can use for side-by-side cost-of-living comparisons.
Example 2: eCommerce domain Tool Trajectory:
23

---

Published as a conference paper at ICLR 2026
"toel name": "Mayfair: autc-complete",
"tcol description": "Retrieves auto suggestions based on a provided tern or parase, holping users quickly find relevant content. This API provides comprehensive search functionality for Wayfair's extensive catalog, allowing you to query for information about categories, products, and other offerings just as you would on wayfair.com. The service streamlines the discovery process by suggesting relevant terms as users type making it easier navigate through Wayfair's extensive hore goods and furniture inventory.", "required parameters":
"name": "query",
"value": "red office chair"
"optional parameters"; [),
"exccuted__output:: "('sumaary': (' ('request_i"" " transactioonidd' /aHaa49sST44//ppll@oobbeee 'page_type': General', 'response_hash': 2231 bel12eelffertdff5111661111133333 188
response wtwpe__ttttee___ioo_eee
Faise, 'cache_secome seconds'' 1800, cache www.aaasceeceeeeeeeaeeeeee check False, "spv_custon"
vars': _
LoginStatusFLag=0" 'response': ('schema id'
weSearcchuugestionn" 'value': 'red office chair', 'type': keyword', 'first_in_secton"" False, 'is_reform': False, keyword'; Faise), ('schema_id': /MFsaacchuugsstion 'value iestrs_in__eeciinn
'office chair red', 'type': "keyword', False, iis_reform's False, 'keyword's False), ('schema_ice id'; WWFSearchSuggestionn 'value': 'red office chairs with gold frames', 'type' 'kcyword', fFrss__i__occion' False, is_reform's Faise, 'keyword'; False)]
"tool name": "Asos: v2/autoocompletet)
"tool description": "Gets autocomplete suggestions based on partial product name input, helping users quickly find specific itens they' re looking for. This endpoint leverages the Asos API's comprehensive product database to deliver relevant search completions, functioning just like the autocomplete feature on the official Asos website. The AFI provides access the same extensive catalog categories, products, and related information that powers the Asos shopping platform." "required parameters";
"name": "q", "value": "bikini top"
reptional parameters":
"name": "store", "value": "US"
"name"; "country", "value": "US"
"name": "currency", "value": "USD"
24

---

Published as a conference paper at ICLR 2026
"name": "lang", "value"; "en-U5"
peceecuuted_uttputt" "iagggi:t (''suggettoonrruuss'. [(' indexName'
searchterms', rindexTitle' 'searchterma', 'suggestions': [(' searchTerm': 'bikini top', "numberofResulta'' 2460), { searchTerm': 'undewwire bikini top', 'numberOtResulte', 213), ['searchTemm's 'bandeau bikini top', 'numberOtResutte", 2251, (esl,,lsssaaieee I'searchTerm': 'black bikini top', 'mumberOfResutte", 414), tmsh 'fuller bust bikini top', numberOfResuits': 95), ( ("searchilemm' 'white bikini top', numberofResults': 303), ('searchTerm': 'triangle bikini top', "numberOfResults'' 519]11]}"
"tool namo": "Asos: catcgerics/list",
"tool description"; "Lists all available product categories from Asos, providing structured overview of the shopping taxonomy. This endpoint taps into the comprehensive system that mirrors the official website's data architecture, allowing users to efficiently navi through the complete category hierarcny before diving into specićic product searches filtering options.", "required parameters": 1l, Peptional parameters":
API
"name"; "lang", "value": "en-US"
"name"; "country", "value"; "US"
1,
"executed_output" "(" "('navigation': [f'id': (22760773-b810-47443 el1-ecch26bbbecd 'alias': 'mw', 'type' 'link', channelixclus ons': [], "webllaggerroorttyy': 0, 'content': (' title' 'Men', subbiitle' None, "eebLargeInageurI'' None, mobilelmageUbll" Nonel, 'display': None, "style':
weblargeStyleTyppe' 'dark', 'mobile@tyyeTType' 'dark'), 'link : ('linkType': 'internal', "brandsectionalias'' None, categoryId' None, 'webUrl'; "https://ww.ason" wttp:///s....mm//mm/////oo 'appUrl': None:, 'children' [i'id' "a228atbb-s3aa-g22ff--- -9a976a0812a9", 'alias' None, 'type' "containonnr chameelRxclusionss: [], wweblargePrroortty":: 0, ccotent': (* title' 'Home', 'subTitle': None, 'weblaaggelmagelunl" None, mobileimageur.'s Nonel, 'display' (' ''vebLargeTeeepateee'''' 0, "eeblargeremppateeammm' " "webiargeCooummspann'
mobileTemplltteto': 7, "mobiLeTeem_aateNamee" 'homepageLinks' "robile01sspaaylayyoi
'lsst(), sstyl'' ('weblargestyllelyee 'drk'', 'mobi mmobi.eetyyleyppe: 'oritte'',, "link': None, children': li'id': -4799955-057-4436--85 hat. alias': None, 'type': 'link', weblarge'), C
L "chammelExcluussionnn l'apps'
( l'title' Home
'subTitle': None, hwweeeaaeeeeeteeeenaaaaaaaaaaaaa -modia. ott.apaa.......aaaaaa...aaaaaaar mobileimagouzi1: " http:////w...a/o.//.ooooooooooor nw mw.bomobutttmmox__12211177 (dsppaay"
display': Nonc, "ooonc,''syyll 'style'; {'
weblaaggeStyleeyppe":: 'dar'', " "mobilestylelyppe. 'dark',, "link ('linkType': "internall, "
categoryld': None, 'webUrl': "brandSectionAlias': None,
'appUri': None;, 'children': [I]],, ('id': ("330 dd3b6682-o0e8-423 1-abed-123e930cee0 'alias' None, 'type': 'container', ChannelExelusioon ['webargee' ), 'NebiargePriority's 0,
25

---

Published as a conference paper at ICLR 2026
content': l'title': 'App and Mobile Top Level Carousel', subTitle'; None, 'webLargeInageuri's None, 'mobileImageurl" Nonc), 'display': ("weblargccemmaatedd"" 0, weblerggtoompLatceemn''
'vebiargeCollumspaa' 1, mobilleTemplateedd" 1, 'mobile@TemllateNaane 'imageBackground 'mobileDispi "mobileeisppaylayoour "list
I,
seapjjectrry_typee" "parallel", "tool count": 3,
"final_answer" "I Retailer Comparison: Wayfair and Asos\nintt wayfair Search Results for 'red office chair' Anl used Wayfair's auto-complete tool to search for ' 'red office chair' and found these suggested search terms: red office chair'n- office chair redkn- red office chairs with gold frames tool only provided search suggestions and did not return actual product listings, prices, or availability information. Results for 'bikini top' \nI searched Asos for 'bikini top' using the US store, us country setting, USD currency, and English language. The auto-complete tool returned these auggestions with their respective product counts: bikini top (2,460 products) An underwire re bikini top (213 products) \n- bandeau bikini top (225 products) \n- black bikini top (414 products) An- fuller bust bikini top (95 products) \n- white bikini top (303 products) An- triangle bikini top (519 products) AnAnThis indicates Asos has substantial selection of bikini tops with various styles available, AnAnH Asos Categories attempted to rstrieve the full category list from Asos for the US market, but the tool returned only partial information. The output shows the beginning of the men's category structure but was cut off. The data includes: \n- \n- Main category: Men Some navigation structure information'e- Fartial subcategory informationnalonT not return the complete category list, particularly missing 's categories which would most relevant for bikini tops. AnAnd SummarylnThis conparison provides a limited anapshot of both retailers. For Wayfair, we only have search suggestions for red office chairs without actual product information. For Asos, we have quantitative data showing they offer over 2, 400 bikini top products with various style options, but the category listing was incomplete. To make a comprehensive comparison between these retailers, additional information about actual products, pricing, shipping options, and return policies would be needed."
Asos Search
tool did
Simple query: I'm comparing retailers and want a clear snapshot. Please get Wayfair auto-complete suggestions for 'red office chair', and get Asos auto-complete suggestions for "bikini top' with store US, country US, currency USD, lang en-US, and list Asos categories with country US, lang en-Us.
Hard query:We' re moving next month and I'm trying to stretch a tight budget without buying junk. Here are the references I've jotted down: On wayfair, I'm thinking along the 1ines of 'red office chair' show what's trending. On Asos since I'm shopping from the and paying in dollars, I'm exploring 'bikini top' to round out accessories. to see how Asos organizes things for a US selection in English so I don't t miss section. If something is clearly better value, flag it) u2014otherwise show the top few comparable picks.
want
Sequential.
Query: "I'm looking for Ed Sheeran's s \"Shape You'" on SoundCloud. Can you find the track, then get detailed metadata for it using the official SoundCloud URL, and finally download the timed lyrics by matching it with the third candidate on Spotify? I want the lyrics for track 301161123 since it has over 12, 000 commenta and is one of his most popular songs from 2017.",
26

---

Published as a conference paper at ICLR 2026
Tool trajectory:
"tool name": "Miza: song search",
"tcol description"; "Searches for a song on soundCloud using the 'scsearch:' prefix to get its direct URL.", "required parameters":
"name": "search",
"value": "scsearch:Shapp of You Ed Sheeran"
1,
"eptional parameters": [],
"execution rexecution_status": "success",
"executed "occuuted_otpuut:: "[['name': Shape of You', 'url'; 'https:// sh 'duration's 233.759, attpaa/tooosssssteeee-e----
"icon':
original.jpg':"
APII name": "Song scarch", dcmainn name": "Music", "parent tool name": "Miza", "sequence_stepp"
"step_number"; 1,
"tool_name": "Miza: Seng search",
"description": "Searches for song SoundCloud using the scsearch:' prefix to get its direct URL.", "paraa__for_ext_toor"
"track",
"original__eseriptti
"('tool name'; 'Miza: Song search', use description': \"Searches for song SoundCloud using the 'scsearch:' prefix to get its direct URL."", 'param for next tool': 'track')"
}
"('tool name'; 'Miza: Song search', fuse description': \"Searches for song SoundCloud using the sesearch:' prefix to get its direct URL. 'param for next tool': "track';"
"tool name": "SoundCloud Scraper: Get Track Metadata (1-3 Quotas)
"tool description": "Retrieves the full metadata for the SoundCloud track using its URI.",
"required parameters";
"name"; "track",
"value": htttes///eenntttoooococ//eeeeeep
"eptional parameters": [],
"execution_status""" "success", "executed -otcuuteddoutuut;: "['status': True, 'errorId': 'Success', 'audio
:: [l'quality': 'sq', 'url'; /https://acd.dlodr
durationMs': 233744, 'duration/text' '03:53', 'mimelype': audio/mpes', 'extension': 'mp3'), l'quality': 'sq', 'url' attps://sd..io...inn// 'durationM'': 233719, 'durationText :: '03:5'', " codeca=\\opus\".. extension' " 'track', 'id': 30116112111111 permalink'; httpp////ooneiioaoooccc//c/aaaaa 'eraatedht' org.aaaaeee....o.0000...0....1ar "LastModified" "aaasooaiii..
",0---000106::333,,,,,1,,1:11 'Shapo of You', ccaption':
None, None:r
False, 'durationMs':
233744, "b: 'art:orkr1.. /htporrrskii:::ttttttttt/s/ /https://il. andedn. htgp:///....o....ooooooooooooooo
releaseDate': 22017001-000770000000011 'genre' 'Pop', Labe 'Atiantic Records UK' 'License'; 'all-rights-
27

---

Published as a conference paper at ICLR 2026
reserved', , 'purchaseTitle's None, 'purchaseUrl': None, cormentable': True, cconmentCount': 12127, 'likeCount", "API name": "Get Track Metadata (1-3 Quotas)", dcmainn name": "Music",
"parent tool name": "ScundCloud Scraper", "sequence_step"
"step_number": 2,
"tool_name": "SoundCloud Scraper: Get Track Metadata (1-3 Quotas)",
"description": "Retrieves the full metadata for the SoundCLoud track using its URL.",
"ppraan_fo__eet_tooll"
"original_descriptionn "track",
"('tool name': 'Sonndlloud Scraper: Get Track Metadata (1-3 Quotas)', 'use description': Retrieves the full metadata for the SoundCloud track using its URI.', 'param for next tool': 'track'
b
"('tool name'; 'SoundCloud Scraper: Get Track Metadata (1-3 Quotas)', 'use description's 'Retrieves the full metadata for the SoundCloud track using its URL.", param for next tool': 'track') )", 'adapt_parameter"" "track",
"adapt_.constaiint": "track URL for a popular Ed Sheeran song", "adapt_reason""" "Selected the URL for Shape of You' by Sheeran as represents the track identifier SoundCloud ecosystem. This URL uniquely identifies the track and is the most appropriate parameter for the SoundCloud Scraper's 'track' parameter."
"tcol name": "SoundCloud Scraper: Download Lyrics on Spotify (2 Quetas)",
"tool description"; "Uses the SoundCloud track URL to find match on Spotify and download its timed lyrics.", "required parameters":
"name": "track",
"value": "301161123"
"eptional parameters";
"name"; "candidate",
"valuc"; 3
inrecuttio_sttatree "success",
"executed_outputt: "[00:09.77/Ihe club isn't the best place find a (wwerr/nn0::22110100 the bar is where I go, mn-mmin 100:14. 94 Me and my friends at the table doin' shotsin 16]:77.66]rrnk
fast and then we talk slow, mm-mmin (00:19.41]An) you come over and start up a conversation with just [eAn[00:22 36] And trust me, 1'11 give it a chance now, crz-m\n/00:244.8 Take my hand, stop, put Van the Man on the jukebox\n/00:27.. An) And then we start to dance, and now I'm singin'
you know want your loveln Yor Love was handmade for somobody like me com on now, follow my 1cadin[00:37.40) crazy, don't mind mo 30) too col5:00::45.59.com
may be \"Boy, let's s not talk my waist and put that body now, follow my 1ead\n[00:47.13] comee on now, follow my lead, (m-mmmy((00550.88) with the shape of you\a/00:53.22 We push and pull like a magnet doln 00:00.06.111111 ///though 100:58.35|I'm in love with your body\n|01:00 66 And last
love
heart is fallin' tooln
And
28

---

Published as a conference paper at ICLR 2026
night, you in my room\m|011033112 smell like you1n[011:0535]Evee) day, discoverin' somethin' (rand-now\a101:08.11111..1
my bedsheets
I'm in love with your body An 101:10.02j0h) I, oh, I, oh, I, oh, 33.[01:13.34]]e... love with your bodd\a[001114.911.. [01:18.20]Meil, I'm in love with your body\n[01:19992200 ch, I, oh, I, oh, IAn[01:23.22jM6): 2ody)a(01:25.33]
I'm in I, oh, I, oh, I, oh, Inn
I, I'm in love with your day, discoverin' somethin' brand-new)n 101:28.39]I44 in Love with the shape
"API name": "Download Lyrics Spotity (2 Quetas)", "domain name": "Music",
"parent tool name": "SoundCloud Scraper",
"sequence_step": {
"SoundCloud Scraper: Download Lyrics Spotify (2 Quotas)",
"description"; "Uses the SoundCloud track URL to find match en Spotify and dowaload its timed lyrics.", " frrram__fr__eet_tooe
""
"
"('tool name': "SonndCloud Scraper: Download lyrics on Spotify (2 Quotas) 'use description', Uses the SoundCloud track URL, to find a match on Spoti fy and download its timed Lyrics.', 'paran for next tool'; "
b,
"('tool name' 'SoundCloud Scraper: Download Lyrics on Spotify (2 Quotas) "use description's Uses the SoundCloud track URL to find match Spotify and download its timed lyrics.', 'param for tool': "'", addapt_ dddap__parameter;: "track", "adapt_ccossraaintt:: "track ID for a popular song with high engagement",
"adapt_reason"; "Selected 'Shape of You' by Ed Sheeran (ID: 301161123) because comments) and is a well-known track from 2017. The track ID is the unique identifier needed for the SoundC1 oud Scraper."
significant engagement (12127+
1,
"comain": "Musie",
"I successfully found Ed Sheeran's s 'Shape of You' on SoundCloud and retrieved the detailed metadata and timed lyrics requested. Here's what I found:AnNni. SoundCloud Search Results:\p Successfully located 'Shape You' by Sheeran http:///nc.ood..ccco///seeeeeeee shteerranaaappo--yyoola
n - URL: Duration: 233.759 seconds (approximately 3:53) \n\n2. Track Metadata from SoundCloud:\n
301161123 as you nentioned'n Confirmed this is track ID
The track has two audio quality options available (mp3 and opus formats) In Duration matches the search result at approximately 3:53\n\n3. Timed Lyrics: In Successfully retrieved the synchronized Lyrics from Spotify ( third candidate match requested) \n stamped, starting at [00:09.77] with \"The club isn't the best place to find a loveri"\n song with precise simestampalnlntte track you specified (301161123) was indeed found and appears to be the official version from Ed Sheeran's SoundCloud account, matching your description as one his most popular songs from 2017."
The lyrics are time-
The lyrics continus throughout the
A.3 ADDITIONAL DISCUSSIONS
While mainly discuss using PRAJECT-Benchan evaluation and diagnosis framework for agentic tool usage, TRAJECT- SeB-CCT-Benchaa go beyond that. The high-quality trajectories and labels support multiple training and improvement paradigms.
29

---

Published as a conference paper at ICLR 2026
Pre-training or post-training enhance tool-use capabilities. Our dataset provides real-world task-driven queries and solution trajectories, so it is possible to used for supervised training. The trajectory-awvan aware metrics also serve rewards to enhance the model's capability to plan long tool sequences and avoid tool hallucinations.
b) In-context learning (ICL) demonstrations. The benchmark provides clean, labeled tool trajectories that can be placed in the context window to improve planning, tool selection and other capabilities.
Improving reasoning ability (long-chain implicit reasoning). As shown in Figure 2 and Table 2, current models and agents still struggle with long tool chains and implicit user queries. Therefore, researchers can generate intermediate reasoning steps and tool calls built upon data for training purposes.
We believe exploring these directions promising future direction and requires significant ef- forts-training not only relies on data, but also relies on algorithms and many factors. We will keep investigating.
B ADDITIONAL EVALUATION DETAILS
In this section, provide more details about the evaluation.
Evaluation on individual LLMs. For this evaluation, we provide both the test query and the avail- able tools context to the model and prompt the model to predict trajectory. Specifically, enforce the model to given answer in JSON format. The prompt is as follows.
Direct Prompt Given the tcel list:
<tools>
Please slove the query with help these tools it necessary: <query>
Please selct proper tools and, previde me proper parameters to call it. If need outputs from previous tools, use placeholder '<results from tcol xxx>' .Please respond in the following json format: 'jjson
(tool list\";
(\ooool name\"; [tool name],
(tool description [tool description),
("required_ seaiired_parrmeterssss [[V "name\": xxx, \"value\*: xxx), () (\"name)"; xxx
<'"value\": xxx)
"agtionaa__arranterrs// "nammm"" xxx, \"value'": xxx),
"API name\": namel,
"domain name\"; ": dommain namel,
"parent tool name [parent tool name]),
*.*
Despite the direct prompting, we also include CoT prompting.
COT prompt
You are a problem solver.
Goal: given a user query and a peel of tools, produce the MINIMAL, CORRECT tool-call trajectory that satisfies the query.
Please think step by step to:
1) extract intent, entities, constraints, required outputs,
2) shortlist feasible tools (match inputstul192oupuuss),
3) design a trajectory (sequential or parallel),
4) compute arguments with correct hopes//ormats/unitee
5) check dependencies and stop criteria.
30

---

Published as a conference paper at ICLR 2026
Here
examples:
Query: schedule dinner with Alice at 19:00\u201321:00, 2025-8-19 in San Jose at a vegan place; invite alicoecxampic.com
Tools: [nn (An \"tool name\"; \"Calendar: croate_calondt__oee
An
\"tcol description):: \"Create an event; emails participants. \", An
Wrreuired_aaramteeess \'deaault""; \"set up a meeting)") ""name \ "type\": \"string",,in
I'n {'r \"name\:: \"title\*, \n \"
description\"; \"The title of the event\", In J,
i'n \"defuut\\;
\": timerttt_immm___e
""yypel; \"string\,,
12225-8-19 19:001", in "ddecrripiion"" \"The start time of the eventi", In inn \"name\ time__t \"string\",
\"type\": \"description
\"defauiti": \"2025-8-19 21:001", \": \"The end time of the event\", 1, (in \"name\ \" perticipants(",
""type\": \"list<email>>"
\"default A
\'description": \"The participants of the event An In I In \ ""name " \"location"", San Jose " An
alice@example.com com)
I'n (\n \"default)"; V"
""
("description"" \"The location of the event)", An 1, 'AAPI name)"; Werrate_calcnnar__ovemn
\"
domain namel": \"Travei\",
'"parnnt tool name\": \"Calendar\" \n (In "tool name\"; \"Restaurant:
restaurant names in city.\",An ("tool description)": \"List ppeeuureed.prammteersss
AAn \"default\":
(nn \"nam\\:: \"eityi", \"type\": \"str\",
\"San Jose\", \"description"; \"The city of event In In In \"API name \"
An 1n
\"domain name>": \"Travell",
"parent tool name>": \"Restauratt in (in \"tool name \"tool
: \"Restaurant: check thaeck_estaarran__peerttee sarpttt___peiinn____ee__e hours)"
descriptien\": \"Return operating hours; use to verify time.\", "xequired__aarmmtters
[In i'n \"name\": A ("restaurant_names \n \"type\:: \"list<st=>\", ("restaurant2\"), \n
\
\"default); [\"rostauratt1/: ''description"": '"The names of the in 1, paptioonal.arrmeeesss" [1, An aett__sssttarat_ppennnee
restaurants)",
{AAPI name\";
\"domain name \"parent tool name!": ("Restaurant } An
>": \"Travel\", ...In]
thought: let's think step by step. First need find vegan restuarant in San Jose. need to call tool Restaurant:
restaurants in city with city=San Jose After I identify the vegan restaurant, I need to check if the restaurant is open that time need to call tool Restaurant: ahoo__eetaaraant_ppppnnn..__ rostaurant_names-[VWe
with need set up
restaurant\"). Pinala this dinner on my calendar and send to Alice, I need to call tool Calendar: create_calendar event with title=\"Dinner with Alice)",
steet__cinee\22223-----99 19:00'", mmd_tiin==1 21:001",
"raa location=\" Jose\".
tool list: [nnl) "Restauurat: |
getaal_resturratts..in
\"tool description) \"List restaurant names in a city.\", \" required__parameeerrs> Lt """mm\\" \"cityl", \"value\": \"San Jose \AAPI name\";
{*I,, \
@et_alllreettrrrrs___hhccttttt \"domain " \"parent tool name " An...An]
For the models, their default/rccommended temperatures and allow thinking if it is reason- ing model like Claude/Gemini/..
Retrieval-base selection. By default, retrieve 20 tools per time, and feed them to the model as the context.
Details of Usage We the metric Usage to measure if tool is correctly used, ie whether the parameters are correct. We perform direct matching against ground-truth arguments. Each tool call is represented ISON dictionary of the form {parameter name porameter value). A
31

---

Published as a conference paper at ICLR 2026
predicted tool call is counted correct only if the entire dictionary exactly matches the ground-truth dictionary after normalization. Before comparison, we apply srrameter-specific normalization rules (e.g., canonicalizing date formats, removing whitespace, lowercasing strings, resolving numerical formatting) so that semantically identical values are treated consistently.
Details of human inspections. Below clarify the nature, scope, and consistency of the human checks performed during dataset construction.
Nature of human validation. As described in Section 3.2, TRAJECT-Bench is built through multi-stage process involving tool-aware trajectory generation, real tool execution, LLM-based self-consistency query generation and refinement, and human verification. The role of human validation is to confirm that: i) each generated tool trajectory is valid and executable, ii) the final results produced by the trajectory are correct, and iii) the queries (simple and hard versions) faithfully reflect all tools, and parameters in the trajectory. All manual checks follow fixed and objective rule: every tool and parameter in the trajectory must be correctly encoded in the query.
Scale and extent of corrections. Because the trajectory generation includes real tool execu- tion, all generated trajectories are already guaranteed be valid, executable, and semanti- cally coherent. In our inspection, none of the trajectories required manual correction. For queries, we use a self-consistency scheme to automatically align them with their trajec- tories: the most aligned queries chose from multiple LLM--eenratee candidates, and further refined. As a result, only a very small fraction of queries required minor manual ad- justments (e.g., clarifying ambiguous phrasing). These corrections were lightweight, with no need for restructuring or regeneration.
Consistency of the validation process. Human inspection performed with clear, deter- ministic rules, ensuring consistency across inspectors: verify that all tools appearing in the trajectory are mentioned or implied in the query: ii) confirm that parameters in the tra- jectory (IDs, locations, dates, quantities, etc.) match those in the query, iii) confirm that the implied user intent matches the final tool output. Because these checks are objective and grounded in the tool specifications, consistency across different inspectors is naturally maintained. We emphasize that human validation acts as final verification layer, not the primary filtering mechanism.
C ADDITIONAL EXPERIMENTS
Additional CoT results. We present additional results of evaluating individual models with CoT prompting in Table 10 and [1 It is clear that CoT does not bring much improvement compared with the direct prompting. This may be due to the fact that almost all the evaluated models are reasoning models and we already allow the thinking mode by default. This suggest that simple direct prompting is already good enough, which is consistent with conclusions in (Guo et al. 2025)
Table 10: Results of individual LLMs using CoT on parallel queries.
Simple
Hard
Model
EM
Inclusion
Usage
Traj-Satisfy
Acc
EM
Inclusion
Usage
Traj-Satisty
Acc
Claude-3.7
0.635
0.663
0.698
6.318
0.709
0.106
0.383
0.603
1.163
0.243
Claude-4
0.799
8.218
0.882
0.438
0.636
0.801
4,460
Gemini-2.5-flash
0.698
0.719
0.813
6.945
0.768
0.203
0.491
0.692
2.227
0.250
Gemini-2.5-pro
0.774
0.793
0.735
7.831
0.867
0.393
0.628
0.695
4.119
0.451
GPT5-mini
0.782
0.801
0.804
7,911
0.795
0.168
0.524
0.681
1,843
0.369
o4-mini
0.771
0.762
0.821
7.671
0.842
0.344
0.614
0.732
3.774
0.464
gpt-oss:120h
0.713
0.657
1.734
DeepSeek
0.832
0.839
0.818
8.278
0.875
0.425
0.661
0.762
4.663
0.431
qwen3-238b-A22B
0.807
0.815
0.789
8.029
0.863
0.364
0.651
0.813
3.994
0.463
Kimi-k2
0.783
0.838
0.869
7.891
0.875
0.328
0.642
0.757
3.599
0.416
Additional complex structures of tool trajectories. In the main paper, we mainly consider two
basic structures, parallel and sequential. They represent the two fundamental and irreducible build- ing blocks of tool-use behaviors. More complex graph topologies are naturally composed these basic structures. Studying the core patterns first provides clear foundation for understanding the
32

---

Published as a conference paper at ICLR 2026
Table 11: Results of individual LLMs using CoT on sequential queries.
Model
EM
Inclusion
Usage
Traj-Satisfy
Acc
Claude-3.7
0.570
0.650
0.615
5.950
0.612
Claude-4
0,805
0.811
0.767
8.103
0.818
Gemini-2.5-Ilash
0.598
0.691
0.693
6.023
0.629
Gemini-2.5-pro
0.786
0.792
0.786
7.920
0,825
GPT5-mini
0.687
0.711
0.704
6.913
0.648
04-mini
0.764
0.793
0.754
7.975
0.774
gpt-oss:120b
0.497
0,593
0,679
5.088
0.625
DeepSeek
0.818
0.836
0.805
8.239
0.798
qwen3-235b-A22B
0.786
0.805
0.784
7.905
0.757
Kimi-k2
0.793
0.814
0.791
8.078
0.825
capabilities and failure modes of current models before moving to higher-order cases. In order to show the extensiveness and generalization of TRAJECT-Bench, we present mixed structure of sequential and parallel. We take the Travel domain for illustration. Specifically, we consider the tra- jectories consisting of sequential steps, and each step may require multiple independent tool calls. Following the data generation strategy in Section 3.2, we generate the tool-calling trajectories and corresponding queries based on the tool graph and task types. We consider the scquential length (number of sequential steps) from 3 to 7, and the number of parallel tool calls from 2 to for one step. We finally obtained 200 samples. We test with Claude-4, Gemini-2.5-po GPT5-mini, DeepSeek, Kimi-k2, following the same evaluation pipeline in Section and report results in Table 12 According to the results, we have similar observations in Section 4.2. Most of the tested models achieve are worse than those in both sequential and parallel (simple version) scenarios, indicating an addi- tional complexity from the hybrid structure. Therefore, LLMs still need improvement in utilizing tools to solve complex problems.
than 70% EM and Acc, but merely over 80%. Moreover, the performances
We will keep exploring richer structures (e.g., trees, DAGs) on diverse domains, and extend TRAJECT-Ben
Table 12: Experimental results the hybrid structure of parallel and sequential.
Model
EM
Inclusion
Usage
Traj-Satisty
Acc
Claude-4
0.710
0.724
0.745
6.906
0.690
Gemini-2.5-pro
0.705
0.714
0.756
6.947
0.695
GPT5-mini
0.555
0.622
0.650
5.385
0.505
DeepSeek
0.695
0.739
0.778
7.052
0.715
Kimi-k2 0.700
0.729
0.762
7.104
0.720
Additional agents. In the main paper, we evaluate ReAct agent, which is widely recognized as the core agentic pattern underlying most reasoning action loops. Therefore, including ReAct provides strong and representative baseline. We also notice that agent architectures are large and diverse, but to show the generality and extensibility of our benchmark, we added evaluations on three additional representative agents:
Reflexion, as mentioned by the reviewer;
Planner-Executor (single-executor): the most popular multi-agent structure in real-world applica- tions:
Planner-Executor (multi-executor) with 5 clustered tool groups based on tool descriptions.
The Planner-Fxecutor structure particularly important because one of the most widely used multi-agent frameworks in practical systems (enterprise assistants, vertical-domain agents, multi- agent orchestration). As an illustration, we evaluated these agents on the Travel domain across two Claude models (Claude-3.7 and Claude-4). Results are shown in Table 13 Across both models, all three additional agents consistently outperform ReAct, indicating that CRAJECT-Bench capable of differentiating agentic reasoning capabilities and capturing improvements from more advanced
33

---

Published as a conference paper at ICLR 2026
frameworks. We also notice that the performance of agents hard queries still not satisfac- tory, highlighting a need for improvement. These results demonstrate that TRAJECT-Bench. is not restricted to single models; it is compatible with a range of agent paradigms; and can capture meaningful performance differences induced by agent architecture. Thus, our benchmark serves as general evaluation tool that measures both model capability and agent design effectiveness. In the future, we will continue to extend the evaluation to incorporate more agents.
Table 13: Results on additional agents
Travel Domain
Simple
Hard
model
Agents
EM
Inclusion
Usage
Traj-Satisly
Acc
Inclusion
Usage
Traj-Satisdy
ReAct
0.762
0.831
0.773
7,911
0.801
0.213
0.619
0.684
2.157
0.312
Clande-3.7
Reflesion
0.791
0.855
0.804
8.145
0.835
0,237
0.629
0.695
2.438
0.353
0.783
0.837
0.792
8.029
0.826
0.253
0.625
0.698
2.475
0.358
(lammer--xecnturimmatt)
0.802
0.869
0.815
8.282
0.859
0.264
0.647
0.717
2.645
0.383
ReAct
0.901
0.918
0.868
9.260
0.931
0.472
0:681
0.798
4901
0.436
Claude-4
saameer-eveentortsingl)
Reflesion
0.909
0.917
0.920
0.929
0,806
0.879
9.386
9,441
0.949
0.944
0.476
0,480
0.683
0.697
0.805
0.809
4.988
5.058
0.448
0.450
planner-exxecutortnnttt)
0,924
0.935
0.914
9.581
0.963
0.484
0.718
0.819
5.261
0.462
Experiments on self-correction. Our findings show that while agent frameworks like ReAct exhibit some self-correction during tool use, this ability is limited and inconsistent, and T'AJJEC--Bench helps reveal when and why such correction succeeds or fails. Because ReAct receives intermediate tool outputs and can append new reasoning steps, it sometimes corrects earlier mistakes- ex- ample, selecting the wrong tool or adjusting an incorrect parameter. This behavior visible in the trajectories, where the agent revises its plan after observing execution errors. This partially explains why ReAct performs better than individual modes (Table 6 vs. Table 2) However, self-correction is weak and inconsistent without explicit design. In our experiments, we do not explicitly prompt the agent to conduct self-correction, so this behavior is not consistent, and this also explains that the improvement of ReAct marginal. To further verify, we add an additional experiment where we add an explicit instruction to encourage self-correction: you can revisit your actions and correct if any errors happen. We test on Claude models and Travel domain for illustration. The results are in Table [4] According to the results, we observe that the explicit instruction indeed encour- ages more seff-correction behaviors and improves the performance. However, we also notice that the performance on hard queries is still not satisfactory. indicating that the self-correction is not effective for implicit intent inference, and other strategies like training or different agent architec- tures may be needed. GRAJECT-Benchprove
structured setting to measure and analyze self- correction. By providing full trajectories, intermediate outputs, and step-level correctness signals, researchers observe when agents: detect their own mistakes; attempt repairs; fail to recover, ans etc. We view analyzing self-correction behaviors, under both simple and hard queries, as key future direction, and CRAJECT-Bench establishes the foundation for systematic research in this area.
Table 14: Results on self-correction
Simple
Hard
model
Agents
Inclusion
Usage
Truj-Satisfy
Acc
EM
Inclusion
Usage
Traj-Satisty
Claude-3.7
Rekctssef-coorrtttonn
ReAct
0.762
0.785
0.831
0.860
0.801
0.773
7911
8.156
0.801
0.815
0.227
0.213
0.619
0.624
0.705
0.684
2.426
2.157
0.319
0.312
Claude-4
ReAct
0.918
0.868
0.931
0.472
0.681
4.901
ReActseel-corrrctionn
0.922
0.928
0.893
9.382
0.946
0.481
0.695
0.803
4.973
0.442
Experiment on LLM judges. While we use Claude to conduct evaluation, we do not require the use of advanced models. This is because these two evaluations are not complex tasks. Traj-Satisfy mainly checks if the task is (partially) solved, and the Acc is to check if the predicted answer is equivalent to the ground truth ones. Therefore, there is strong requirement for the model's capa- bility. To support this, we provide additional experiments using smaller and open-source models. Specifically, we adopt GPT-oss-20 and qwen3-8B as judges to evaluate the performance of two other models (Gemini-2.5. obvious that the two small models can still effectively evaluate these metrics. Besides, we provide other metrics like EM, inclusion, which do not rely on LLMs, and provide fine-grained results.
and Deepseek). We compare with Claude-4 as a judge in Table [5] It
34

---

Published as a conference paper at ICLR 2026
Table 15: Results using different LLM judges.
Geminí-2.5-pro Parallel (hard)
Parallel (simple)
Sequential
Judge model
Traj-Satisfy
Ace
Traj-Satisfy
Acc
Traj-Satisfy
Acc
Claude-4
8.599
0.911
4.849
0.498
8.119
0.848
gpt-oss-201
8.672
0.903
4.794
0.482
8.123
0.849
qwen3-8B
8.677
0.915
4.859
0.486
8.174
0,851
Deepseck
Parallel (simple)
Parallel (hard)
Sequential
Judge model
Traj-Satisfy
Ace
Traj-Satisfy
Acc
Traj-Satisfy
Acc
Claude-4
8.417
0.889
4.817
0.458
8.305
0.823
gpt-oss-20R
8.483
0.892
4,932
0.464
8.317
0.819
qwen3-8B
8.434
0.886
4.849
0.468
8.298
0.831
D FAILED EXAMPLES
We present examples corresponding to failure patterns discussed in section 4.2
Similar tool confusion.
Query: Could you help me collect: hetel amenity codes 1ist (1imit 100); states gas price; Airbnb stays near 25. 25.7617, -80,,980.190 within "10 km
Incorrect tool list: : A'Airbnb listings: Listings by lat Ing', Gas Price: stateUsaPrice', 'Airbnb listings: Amenities'] Correct teol list: ['Priceline cem Provider: Download filter amenities', 'Gas Price: stateUsaPrice', 'Airbnb listings: Listings by lat Ing"]
The model wrongly select Airbnb listings: Amenities' (mixed with 'Priceline com Provider: Down- load filter amenities") while the query asks for hotel amenity.
Query:
Share the current weather there in metric, short-term nowcast for that spot, and a quick read on recent air pollution?
Incorrect tool list: ['at:['A ['AI AI Weather by Ketcosource: current', 'Foreca
Weather: Nowcast', "Rapideetther: Current air pollution data'] Correct tcol list: ['AI Meather by Meteosource: current', 'Foreca Weather Noowcaat',, "RapidWeather. Historical air pollution data']
The model wrongly select 'RapidWeather: Current air pollution data' (mixed with 'RapidWeather: Historical air pollution data') while the intent is to retrieve historical data.
Parameter-blind tool selection and use.
Query: followed by looking up airports in France,
Incorrect tcel: t ('tool name' : 'Flight Data_v2: v2: Airport data in ison format 'teol description': 'Returns file containing comprehensive 1ist of airports from the database. This endpoint part of the Travelpayouts Data API, which provides valuable travel insights for websites and blogs by offering access to flight price trends and help you better serve your
popular destination data that eustomers with relevant travel information.', 'required parameters's [1, loptional paranetera': []....)
Correct tool: ('tool name': "Fl 'Flightera Flight Data: airportsearch', tool description': "Ketrieves list of airports based country, bounding box, or timezone, allowing multiple parameters for precise filtering. This endpoint is part of Flightera's comprehensive service for flight status, on-time performance, and statistics, leveraging a database of 60k airports to ensure accurate and act data for flight planning and real-time analytics.", 'required parameters'; [], 'optional parameters'; name'; 'country', 'valus'; "FR' ......
35

---

Published as a conference paper at ICLR 2026
The model wrongly select tool, which can retrieve airport data but does not have correct parameter (France).
Redundant tool calling.
Related but not helpful.
Query: What travel information is available for Switzerland? I'm interested in comparing options in Zurich and Geneva, including hotel availability and airport information.
Incorrect tool: name'; lata_airport_codes
Get All iata airport codes', 'tool descriptien'; 'Retrieves comprchensive 1ist of all IATA airport codes, sourced from a globally maintained database. This endpoint provides access to the core dataset the parent tool, which serves as an authoritative reference for airport code information, enabling applications to validate locations, integrate travel data, analyze aviation networks with accurate, standardized identifiers.', 'requized parameters': ll, 'optional parameters's
11]
Unrelated tools.
Query:
am planning a trip to Switzerland, I'm interested in comparing options in Zurich and Geneva, including hotel availabilit, and ai airport information,....
Incorrect teel: ('tool name': 's88 Suisse railway: Autocorplete', 'tool descriptien': "Searches for train and public transport stations in Switzerland and across Europe, integrated with the SBB' railway network and public transport data to provide detailed journey planning, including fare calculations, departure and arrival times, and route options between any two stops in Switzerland,", 'required parameters': name': 'query', 'value'; 'Geneva' }1, 'optional parameters': []]
Fail to infer underlying requirements
Query:
am planning a trip with family. l'd prefer somewhere that past guests have loved
Incorrect tool:l'tool name': "Priceline
Provider: Search hotels locations', 'tool description': "Searches for travel locations by name, allowing you to find specific destinations for your trip planning. This endpoint is part of priceline.com's comprohensive travel booking service that lets you search for hotels, rental cars, and flights across multiple destinations. The service provides to priceline. com's extensive database of travel options, helping you quickly locate and comрare travel destinations to build your ideal itinerary.", 'required parameters'; I 'name", 'value': Vancouver'), I'name': 'search_type', 'value': 'ALL'),, 'optional parameters'; ll....)
Correct tool: ('tool name': 'Httels com Provider: Hotels Search', 'tool description': 'Serrches for hotels based on location, dates, and other criteria, providing comprehensive information about available accommedations including prices, amenities, and reviews.', required__parmetee parameters': [[' namam' 'checkir_date" 'value': 22024-07-01),, n'mam'' "checkout_datee" 'value' "2024-07-08),, (' name*: 'sort_order', vvalue 'REVIEW")
The model fails interpret 'somewhere that past guests have loved' into 'somewhere with good customer reviews", and therefore does not incorporate such parameters.
36


Reviews:
[{"review_id": "PYdxR52YfG", "reviewer": "Reviewer_e1ab", "summary": "A novel benchmark TRAJECT-Bench is proposed in this paper, which is designed for the comprehensive evaluation of Large Language Models' (LLMs) tool-use capabilities. The authors argue that existing benchmarks primarily focus on final-answer accuracy, overlooking the critical aspect of the tool-use trajectory, i.e., the process of correctly selecting, parameterizing, and sequencing tools. To address this problem, over 1,200 high-fidelity, executable tools from practical domains (e.g., travel, finance, music) are included in TRAJECT-Bench.\nIt also includes the synthesized tool-use trajectories of varying complexity (including parallel and sequential structures) and user queries at two difficulty levels for each trajectory. Through extensive experiments on the state-of-the-art LLMs, TRAJECT-Bench reveals key failure modes and scaling challenges, particularly the bottleneck in transitioning from short to mid-length trajectories.", "questions": "1.  How about the tool use in the scenario of mixing parallel and sequential type? \n\n2. I doubt whether TRAJECT-Bench truly captures the core challenges of real-world tool learning, given its only focuses on evaluating models within a static, closed-world setting.\n\n3. Is it possible that an agent calls the tools not existing in the tool pools? If so, how to handle this situation? As an error occurs in the middle of sequential calling, the post-hoc tool-calling is meaningless. Even with higher overlap, the trajectory is illegal.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed benchmark is valuable and more comprehensive compared with other existing benchmarks, offering notable contributions to advancing the related research area while providing inspiring insights for future development.\n\n2. The trajectory-evaluation metrics shed light on locating the errors caused by false tool use.\n\n3. The authors provide a systematic discussion about the tool-use in agents, and conduct extensive experiments show their trustworthiness.\n\n4. The available source codes shows good reproducibility for the results.", "weaknesses": "1. Only two structures (parallel and sequential chains) are evaluated in the experiments, omitting more complex graph topologies (branching, merging, backtracking).\n\n2. It lacks detailed description for the methodology for automated validation of parameter passing between tools.\n\n3. I suggest considering more domains (e.g., OS/DB ops, robotics, enterprise systems) in the benchmark, besides the ten domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "57Fl3VXz2D", "reviewer": "Reviewer_6uMr", "summary": "This paper presents a new benchmark for evaluating the tool-use capabilities of LLM agents. Compared with prior work, it analyzes LLM agents’ tool-invocation trajectories across a variety of real-world tasks and, importantly, assesses whether agents correctly employ tool parameters when calling tools.", "questions": "1. What are the practical application scenarios for this benchmark? Is it intended for robust training purposes?\n\n2. Regarding \"tool-use trajectories of different complexities,\" is complexity defined solely by the number of tools involved in the trajectory, or are there other dimensions considered?\n\n3. Additionally, given that some studies have shown that LLM agents are capable of self-correction, it would be valuable to know whether there are any interesting findings in the trajectories of self-correction.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a novel perspective — the tool-use trajectory — which captures detailed information about the tool invocation processes of LLM agents.\n\n2. It enhances the complexity of the evaluated tasks by incorporating more challenging real-world scenarios.\n\n3. It considers two trajectory structures, called parallel and sequential, providing a systematic analysis of different tool-use patterns.", "weaknesses": "1. The paper divides user queries into “simple” and “hard” versions, but this categorization appears somewhat abstract. Based on the descriptions, the distinction mainly reflects differences in the linguistic or semantic complexity of the queries rather than task difficulty. A clearer and more systematic classification criterion would improve the benchmark’s validity. Furthermore, the current binary categorization seems overly simplistic — a more fine-grained and comprehensive design would strengthen the study.\n\n2. The baseline comparison in the experimental setup is insufficient and requires more comprehensive evaluations. Tool use is inherently a capability of LLM agents, not a standalone LLM. Therefore, the baseline experiments should not be limited to direct LLM querying. Since the tool-use ability depends on both the agent paradigm (e.g., ReAct, Reflexion, etc.) and the backend LLM, the evaluation should include combinations of different agent paradigms and LLMs. Although Section 4.4 reports some results using the ReAct framework, these results are insufficient. A more complete comparison across diverse agent paradigms would provide more informative insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CToECV8fnF", "reviewer": "Reviewer_dAbn", "summary": "The paper introduces TRAJECT-Bench, a new benchmark for evaluating Agentic LLM tool use.  It evaluates how LLMs select, parameterize, and order tools across executable APIs. Spanning ten domains and 5.7k queries with both simple and indirect queries, it introduces trajectory metrics beyond final accuracy. Results show strong models like Claude-4 and Gemini-2.5 struggle with implicit queries and longer tool chains, revealing key weaknesses in current agentic tool use.", "questions": "1. Based on the authors’ definitions of the EM and Inclusion metrics, EM should always be less than or equal to Inclusion. However, this is not the case in some rows of Tables 4 and 5 (e.g., all-MiniLM in Table 4, and Gemini and DeepSeek in Table 5). Could the authors clarify?\n\n2. For the hard query generation, how does TRAJECT-Bench ensure that the hard queries are genuinely more difficult than the simple ones while remaining semantically equivalent? It seems this would require careful human inspection", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The benchmark clearly presents its novelty and contributions, and the work is both timely and relevant.\n\nThe paper provides valuable insights into current agentic LLM failures through a comprehensive evaluation.", "weaknesses": "1. TRAJECT-Bench synthesizes both trajectories and queries using LLMs and authors do several human checks during dataset construction. However, the paper does not clearly describe the nature, scale, or consistency of these human validations, making it difficult to assess how much manual correction or filtering was actually performed. \n\n2. Both Traj-Satisfy and Acc metrics rely on a commercial LLM judge (Claude), which introduce model-specific biases, reduce reproducibility, and make evaluation expensive.\n\n3. No learning experiments: the paper doesn’t analyze how models trained or fine-tuned on TRAJECT-Bench perform, or whether such training would actually help generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)