Title: TravelPlanner: A Benchmark for Real-World Planning with Language Agents
Benchmark: TravelPlanner
Summary: TravelPlanner is a benchmark created to evaluate the real-world planning capabilities of language agents, particularly in the context of multi-day travel planning under complex constraints. It provides a static sandbox environment with tools and includes 1,225 annotated user queries with reference plans. Experiments reveal that current LLMs struggle significantly with the complexity of these tasks, highlighting major limitations in current agent planning capabilities.
Contribution: "Agents struggle to align their actions with their reasoning" (page 8)
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: planning, tool use, reasoning, multi-constraint planning - capability to plan multi-day travel itineraries by using external tools and satisfying environmental constraints.
Phenomenon defined by authors: Yes
Phenomenon definition: "Planning a multi-day itinerary is inherently long-horizon, involving a large number of interdependent decisions on places, lodging, transportation, dining, etc. [...] requires strong agency to proactively acquire necessary information using various tools [...] and deliberate over the collected information to further the planning while being mindful of all the explicit and implicit constraints."
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Agents are asked to generate travel itineraries that satisfy user-defined and "commonsense" constraints, using a set of external tools to gather necessary information from a simulated environment.
Task item definition: origin, destination, days of travel, dates of travel, query for the agent, level, reference_information
Task definition detail: The benchmark includes both a full planning mode (with tool-use) and a simplified planning-only mode, allowing granular evaluation of agent capabilities​.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 1000
Dataset size notes: 45 training, 180 validation
Dataset metadata available: Yes
Dataset metadata detail: duration, difficulty level (easy/medium/hard), number of cities, constraints
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: Mean and breakdown by category
Metric subscores: Yes
Metric subscores detail: Scores are given separately for delivery rate, commonsense constraint pass rate, hard constraint pass rate, and final pass rate​.

Details from the article:

Delivery Rate: assesses whether agents can successfully deliver a final plan within a limited number of steps. Falling into dead loops, experiencing numerous failed attempts, or reaching the maximum number of steps (30 steps in our experimental setting) will result in failure.

Commonsense Constraint Pass Rate: comprising eight commonsense dimensions, this metric evaluates whether a language agent can incorporate commonsense into their plan without explicit instructions.

Hard Constraint Pass Rate: measures whether a plan satisfies all explicitly given hard constraints in the query, which aims to test the agents’ ability to adapt their plans to diverse user needs.

Final Pass Rate: represents the proportion of feasible plans that meet all aforementioned constraints among all tested plans. It serves as an indicator of agents’ proficiency in producing plans that meet a practical standard."
(page 5)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: No
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Authors analyse failure cases and testing whether task success correlates with clear sub-capabilities (like tool-use and constraint tracking). They also validate the difficulty by comparing human and model performance and show inter-annotator agreement on constraint satisfaction.
Taxonomy: Reasoning / Planning
New taxonomy: General Capability / Planning
Procedural notes: GPT-4-Turbo passes with only 0.6% of tasks, all the other models failed at 0.0%. So overall very low results and often models could not tackle succesfully anything at all.