ChinaTravel: An Open-Ended Travel Planning Benchmark with Compositional Constraint Validation for Language Agents
Authors: Jie-Jing Shao
Bo-Wen Zhang
Xiao-Wen Yang
Baizhi Chen
Siyu Han
Pang Jinghao
Wen-Da Wei
Guohao Cai
Zhenhua Dong
Lan-Zhe Guo
Yu-Feng Li
Venue: ICLR 2026 Poster (2026)
Keywords: Benchmarking
Travel Planning
Neuro-Symbolic Learning
LLM Planning
Primary area: datasets and benchmarks

Abstract: Travel planning stands out among real-world applications of \emph{Language Agents} because it couples significant practical demand with a rigorous constraint-satisfaction challenge. However, existing benchmarks primarily operate on a slot-filling paradigm, restricting agents to synthetic queries with pre-defined constraint menus, which fails to capture the open-ended nature of natural language interaction, where user requirements are compositional, diverse, and often implicitly expressed. To address this gap, we introduce \emph{ChinaTravel}, with four key contributions: 1) a practical sandbox aligned with the multi-day, multi-POI travel planning, 2) a compositionally generalizable domain-specific language (DSL) for scalable evaluation, covering feasibility, constraint satisfaction, and preference comparison 3) an open-ended dataset that integrates diverse travel requirements and implicit intent from 1154 human participants, and 4) fine-grained analysis reveal the potential of neuro-symbolic agents in travel planning, achieving a 37.0\% constraint satisfaction rate on human queries, a 10$\times$ improvement over purely neural models, yet highlighting significant challenges in compositional generalization. Overall, ChinaTravel provides a foundation for advancing language agents through compositional constraint validation in complex, real-world planning scenarios. Project Page: https://www.lamda.nju.edu.cn/shaojj/ChinaTravel/index.html

Published as a conference paper at ICLR 2026
CHINATRAVEL: AN OPEN-ENDED TRAVEL M
PLANNING BENCHMARK WITH COMPOSITIONAL
CONSTRAINT VALIDATION FOR LANGUAGE AGENTS
Jie-Jing Shao'; Go-Wen Zhang Xiao-Wen Yang" Bai-Zhi Chen',
Si- Yu " Haag-Hao uue Pang" Wen-Da Weit., Guohao Cai*,
Zhenhua Dong', Lan-Zhe Guol:;; Yu-Feng Li!3i
State Key Laboratory Novel Software Technology, Nanjing University, China
School of Intelligence Science and Technology, Nanjing University, China
School of Artificial Intelligence, Nanjing University, China
4 NNah's Ark Lab, Huawei, China
{shaoj y, zhangbw, yangxw, guolz, 2, liyf Salyffeellann @lamda - nju eeu.ad..cnunuuuur
ABSTRACT
Travel planning stands out among real-world applications of Language Agents be- cause it couples significant practical demand with a rigorous constraint-satiisaction challenge. However, existing benchmarks primarily operate on a slot-filling paradigm, restricting agents to synthetic queries with pre-defined constraint menus, which fails to capture the open-ended nature of natural language interaction, where user requirements are compositional, diverse, and often implicitly expressed. To address this gap, introduce ChinaTravel, with four key contributions: 1) a practical sandbox aligned with the multi-day, multi-POI travel planning. 2) a com- positionally generalizable domain-specific language (DSL. for scalable evaluation, covering feasibility, constraint satisfaction, and preference comparison 3) an open- ended dataset that integrates diverse travel requirements and implicit intent from 1 154 human participants, and 4) fine-grained analysis reveal the potential of neuro- symbolic agents in travel planning, achieving a 37.0% constraint satisfaction rate on human queries, a 10x improvement over purely neural models, yet highlighting significant challenges in compositional generalization. Overall, Travel pro- vides a foundation for advancing language agents through compositional constraint validation in complex, real-world planning scenarios.
Project Page
Codebase
Dataset
INTRODUCTION
A long-standing goal in Al is to build reliable, general planning agents that assist humans in real- world tasks. Recent advances in LLMs have sparked the rapid development of Language Agents, which employ LLMs perceive the surroundings, reason solutions, and take appropriate actions, ultimately building autonomous planning agents (Shinn et al. 2024 Yao et al. 20236 Xi et al.) 2023) Jimenez et al., 2024). Among numerous real-world tasks, travel planning stands out significant domain, presenting both academic challenges and practical value due to its inherent complexity and real-world relevance. Beyond the travel community itself, such planning scenarios have also become natural testbed for general constraint-awarr planning and reasoning. thereby attracting growing interest from the broader Al community [Kambhampati et al. 2024, Chen et al., 20255] Choi et al.. 2025). Specifically, given query, agents require information integration from various tools (e.g.. searching for flights, restaurants, and hotels) to generate a feasible itinerary. This involves making interdependent decisions across multiple aspects such as spatial, temporal, and financial dimensions, all while meeting the user's 's requirements and preferences (e.g., budget, dining habits, etc).
To assess whether language agents meet users requirements in travel planning, Zhhnng al. present the Trip Planning benchmark for intercity itinerary conditioned on flights information. Xie
2024
"Equal contribution
Corresponding authors

---

Published as a conference paper at ICLR 2026
Ownry
Plasaky
Hiccrnn Itas
Gur budipt $1,80 beitks
Teaphired Syutheds Dura in Prerinas Work
uee skkagghe anw and world like togets sudu aote lecil cuistie My budiptr is 5 sonm wep many
Yast wae cxmusei....
Teala, G114
ence recesnalltten aattes be frieeddredddy Wi une bi like
Skauph lirauiio Jabray Stnker
"Th The Filos Masuu
Jtaliun ressemeen We prster not sell dhitee
Noaeed Massur...
paslilde, paodlppaa
Meno 12:1 1,000 car A vakiigg loke
that nee priteicedl and woald prrtee to bave cntire
$5.900
Phae
The Pilerr Nlacen
DDkkey -xackaai
1030 12.20 pott
fles with
Wakkin 30 11445 wallagg
The dgrio engis 9:00
Deppoot
GLDE Miiiaal
Budlue 5800
sgg Hiatu 17:0 grd
The datiiechuttin
lefox 15PM
Compooltena Conaralia
Tool the
Ieteenatio
The usea cite tee sppertatiion cous 2oth 538, lcaiig budget
2g
Yaxl
Your the Circa Malled Br aooued Sie
hoplet
Weephophog
Viar du her aacuuaaaa at Nariia
Arts Ammurtenioamtta
Nodommationco
Tru Trandowwin))
mone atrracto profite aquua
Test
Wannhilaa
liginry feal Triecing with dics
piarialltiio and contestal ressaiing Haptlighting terengrreeet resl world applestioss
Figure 1: Overview of ChinaTravee Given query, language agents employ various tools to gather information and plan a multi-day multi-PO. Polti-POIItee itinerary. The agents are expected to provide a feasible plan while satisfying the logical constraints and preference requirements. Crucially, ChinaTravel is designed to facilitate the shift from rigid slot-filling paradigms to open-ended natural language interaction, challenging agents to handle the diverse, compositional logic inherent in authentic human intent. To provide convenience for global researchers, we provide the English version here.
et al. (2024) provide pivotal benchmark, TravelPlanner (Xie et al., 2024), with a real-world travel sandbox and various tools to intergrate multi-dimensional information. However, critical limitations persist in these benchmarks stemming from their reliance on a slot-filling interaction paradigm, where agents merely extract values for fixed attributes (e.g.. budget, date), ignoring the compositional logic inherent in human cognition (Fodor 1975 2008; Piantadosi et al. 2016), highlighting a substantial research gap toward LLM Agents capable of genuine natural- language interaction. This effectively restricts agents closed set of intents, highlighting substantial gap toward LLM Agents capable of genuine, open-ended natural language interaction. The limitations arise in: i) Task Bias: favors intercity itinerary while omitting intracity scheduling, where complex, interdependent constraints are desirable. ii) Inflexible Constraint Verification: relies on fixed rule lists. which cannot generalize diverse, unseen requirements spanning compositional concept space. iii) Synthetic Query Construction: includes only templated -Ly-shnteesized queries, underrepresenting open-ended, semantics-rich human requests. iv) Misleading NeSy Evaluation: emphasizes I. LLMM-nll shortcomings in constraint adherence, yet largely ignores neuro-symbolic methods that couple neural language understanding with verifiable symbolic reasoning. Within months of TravelPlanner's ' release, HHoo et al. (2025) proposed neuro-symbolic pipeline: an LLM extracts constraints from templated queries and a formal verification tool yields plans, achieving 97% success rate vs. 4% for LLM-only baseline. This suggests templated, fixed-constraint setups near saturation, failing to expose the true bottleneck of NeSy methods from natural language interaction.
To address the gap. introduce CriveTraveel an open-ended travel planning benehmark. It con- (multi-POI) itineraries (as illustrated in Fig. D and supports the compositional constraints evaluation with authentic Chinese travel queries. It is more realistic and challenging, providing a desirable testbed for real-world travel planning. The main contributions are:
centrates on multi-point-of-intee
We introduce real-world sandbox with a suite of tools aligned with the ubiquitous multi-day multi-POI itinerary planning. It provides the detailed travel information and supports the integration and planning of spatiotemporal schedules.
Sandbox:
Compositional Constraints Evaluation: We present a domain specific language that programmati- cally composes atomic concepts of travel attributes across spatial, temporal, cost, and type dimensions to express compositional constraints, It supports scalable requirement specification and automated constraint verification, with metrics for feasibility, constraint satisfaction, and preference ranking.
Open-Ended Travel Dataset: Beyond the data synthesis pipeline as previous benchmarks, China- Travel integrates human- authorrd queries to create realistic travel planning scenarios. The validation set contains 154 human queries with combinatorial constraint requirements absent from synthetic data, while the test subset provides 1000 open-scenario queries. This structure specifically assesses agents' generalization capabilities for unseen constraint composition.
Empirical Analysis of Neuro-Symbolic Agents: Extensive experiments are conducted and the results reveal that neuro-symbolic agents significantly outperform pure LLM-based solutions on constraints satisfaction, achieving a success rate of 37.0% 37.0% compared to 2.60% by purely neural methods, thus highlighting their promise for travel planning tasks. We also identify the key challenges of worn-world requirements: open contextual grounding, and unseen concept composition, providing foundation for advancing reliable agents toward real-world applicability.
2

---

Published as a conference paper at ICLR 2026
Table I: ChinaTravel's Domain-Specific Language (DSL) for logical constraints.
Name
Syntax
Description
variables
2, U, not eapr
Variables that refer to activities in the travel planning domain.
not
The negation of an Boolean-valued expression.
and,or
expri and expra
The conjunctionkllsjjnntiion of Boolean-valued expression.
<,>.== -,+
expri < expra
Return an expression with built-in number comparison functions.
eupri + expra cost(var)
Return an expression with built-in number calculation functions. A function that takes activities as inputs and returns the attributes, such as cost, type time.
attributes
relation effect union, inter, diff enumerate when
dist(expri, (xppp)))r)rpra))
function that takes locations as inputs and returns the distance. An assignment affects variable ear with the expression expr.
var eapr umi((verts, {war)))
Return set with the built-in union//ntersection/ifffeeee operations of given two sets.
for var in {var)
Enumerate all variables in the collection (var).
if seepr:: effect
The conditional effect takes a Boolean- valued condition of ex- pression exprr, and the effect effect.
Arriving in Shanghai should before PM secord day.
The number attractions visited
Dining expenses 18R8 CNY. dtniing act_ allactivitiessplan):
return_tiee
act_t alLaactwvttiessppann:
act ( in day activitiestplan, 21: typ actlwity_typeeact. type(act_
typ activity, [ppe(act_L] If "brakaaaaa or tynnh"uncc"
setiiityy_tppeaatt ()===/tttatttton count count return count
dest transport des.teeetionneeeeec iontact_ ()
typ=="dtnner"; dining cost
L (types"tratn" or typ "strplane")
dtaiing cost activity castlact return dining_cost
and dasee "Shanghai:: return Fine
Corpare the return during evaluation
activity_ende www.eennneeaat.... actt_.
preference ranking
return, "18:09"
(a) Dining expenses.
(b) Arrived Time.
(c) Count of attraction visited.
Figure 2: Examples of DSL expressions for logical constraints and preference ranking.
Overall, ChinaaTrave provides a challenging benchmark that rigorously assesses constraint satisfaction for travel planning, serving bridge between academic research and practical applications.
CHINATRAVEL BENCHMARK
Motivated by China's 's substantial travel demand, ChinaTravel provides sandbox environment for generating multi-day itineraries with multiple POIs within specified cities. It is meticulously designed provide comprehensive and scalable evaluation framework in travel planning, encompassing three critical dimensions: environmental feasibility, constraint satisfaction, and preference comparison.
2.1 ENVIRONMENT INFORMATION
ChinaTravel provides sandbox with real-world travel information. We collect information from 10 of the most popular cities in China. It includes 720 airplanes and 5,770 trains connecting these cities, with records detailing departure and arrival times, origins, destinations, and ticket prices. Additionally, the dataset contains 3,413 attractions, 4,655 restaurants, and 4,124 hotels, each annotated with name, location, opening hours, and per-person prices. Type annotations for these POIs are included to meet user needs. For a realistic interaction, simulate the API interface of real market applications to query real-time information. We present 25 environmental constraints grouped into six categories: dietary, accommodation, transportation, temporal, spatial, and attraction-related. It acts feasibility metric, ensuring that the generated plans are both valid and effective. For example, POIs in the plan must exist in the designated city, transportation options must be viable, and time information must remain accurate. See App. D.I for design details of sandbox and environmental constraints.
2.2 LOGICAL CONSTRAINT
crucial ability for travel planning is to effectively satisfy personalized user needs. Prior bench- mark (Xie et al. 2024) evaluates logic with five fixed concepts (total budget, room rules, room types, cuisines, transportation types), where each concept is mapped to a specific requirement. Although it has gained much attention, it effectively confines constraint satisfaction to propositional logic, where extracting constraints from mmplate-synthesiin queries is relatively straightforward. In this setting, the system reasons about truth relations between atomic propositions without examining
3

---

Published as a conference paper at ICLR 2026
the complex internal structure or relationships of the travel events. For example, it cannot express that "dining budget is 1000 CNY" or that "arriving in Shanghai should be before PM on day 2", despite the generated itinerary already including the expenses and time information of each activity. Each new logical requirement necessitates human intervention for incremental definition and validation. It is desirable to extend the constraint design and validation into combinatorial language space, which can combines and validates predicates to enable expressive requirements over travel events. We address this gap with a DSL-based solution that enables compositional specification and validation of logical constraints. The proposed DSL provides a small set of basic concept functions and Python-like syntax, so diverse requirements can be written as compositions of primitives and automatically perform validation of plans using a Python compiler. Fig. 2a and [bb illustrate how the DSL, express the user requirements (see Tab. T0 for basic concepts and App. D.3 for a hand-on tutorial with more examples). This approach removes the need for per-requirement rule engineering and yields scalable evaluation of compositional logical constraints from open-world travel planning.
2.3 PREFERENCE REQUIREMENT
Travel requirements encompass not only hard logical constraints but also soft preferences. The term "soft" implies that these preferences cannot be addressed as binary constraint satisfaction problems, instead, they involve quantitative comparisons based on continuous values. This distinction highlights the unique nature of preference- based requirements compared to logical constraints. Common prefer- ences from our surveys include maximizing the number of attractions visited, minimizing transport time between POIs, and visiting positions near the specific POI. In ChinaTravel, we formalize such preferences as minimization or maximization objectives via our DSL, thereby providing an automated evaluation. Fig. 2c illustrates maximizing attractions visited, more examples appear in App. D.6
2.4 BENCHMARK CONSTRUCTION
Stage I: Manual design of database and APIs. We collect travel information for multi-day, multi- POI itineraries across attractions, accommodations, and transportation. We define essential POI attributes (e.g.. cuisine types, hotel amenities) and build structured database from public information. APIs are designed to support agent queries via regular expressions and modeled after commercial APIS to ensure realism. See App. D.I the details of databse.
Stage II: Automatic data generation with LLMs. We model travel tasks with core parameters (origin, duration, etc.) and logical constraints. For scalable generation, we randomly construct query skeletons converted to natural language via DeepSeek- V2..5. Queries are stratified by complexity: Easy (1 extra constraint), vs. Medium (3-5 constraints), with LLM -LM-generated varying expressions (encouraging "Taste Beijing cuisine"-> "Try local food"). See App. D.4 for synthesis details.
Stage III: Quality control and auto-validation. To ensure data quality, we manually check if the generated query conform symbolic skeletons, and re-calibrate natural language description that contain ambiguities. Based on the symbolic skeletons, we verify if the plan can pass the required logical constraints by executing the DSL code via Python compiler. Building on this, we ensure that each query has at least one solution that satisfies the logical constraints via heuristic search.
Stage IV: Open requirements from humans. After the first round of closed-loop development, data generation and annotation, baseline development, and evaluation, we gathered over 250 human requirements via questionnaires. Rigorous quality control yielded 154 queries with novel constraints (e.g., departure time/dining cost), constructing the Human- val- idation set with DSL-annotated automated evaluation. Subsequent scaling through WIX (survey platform) yielded the Human- 1000 test set after analogous quality control and DSL annotation.
including
BENCHMARK CHARACTERISTICS
This section analyzes the challenges instantiated by ChinaTrravel, rooted in authentic human requests and central to real-world applications yet overlooked by prior travel planning benchmarks.
Context-Rich Long-Horizon Planning. China poses unprecedented contextual complexity compared to existing benchmarks, TripPlanning (Zheng al. 2024) and TravelPlannerr (Xie et al.)
4

---

Published as a conference paper at ICLR 2026
600M
5K
Iee
Inper Tokens
ChisaTrwel
4K
ChinsTravel
ChauTravel
Val
200M
10M
TraveiPtamner
Impethrning
2K:
3K:
[K]
Titgetarming
TraveiPlamer
0.2
0.4
TraveiPlazner Trip Planring
Thin
Trin
Val
TravelPlumerr
ChinsTovel
Trlp Planning
10K
50
240
1200
10
10
20
10
50
90
110
POL Count
Day Count
Constramts
Cunsirainis
(a) Token count across different benchmarks.
(b) Constraints across different benchmarks.
Figure 3: (a) ChinaTravel's 's fine- -na-gra spatiotemporal planning demands extremely larger input/out- put text volumes than existing benchmarks. posing fundamental challenges to text-wise planning. (b) ChinaTravel's s authentic requirements, with combinatorial scalable constraints formulation, sys- tematically surpasses conventional closed-form benchmarks in diversity and openness.
2024). As quantified in Fig. 3a (1) Processing over 1,200 candidate POIs per query (4x TravelPlan-
ner's 244 max, 120x Trip Planning's 's 10) with detailed square-order transportation. (2) Generating
540M contextual tokens from dense POI networks, surpassing both DeepSeek-V V3 (64K) and GPT-4o
(128K) capacities, even aggressive 6-POI downsampling retains 40K tokens (Fig. 3ap. (3) Producing
4.8K output tokens for 5-day plans, versus 0.9K (TravelPlanner's 7-day) and 0.5K (Trip Planning's
30-day). These findings necessitate a paradigm shift: the traditional single-past paglee-asssaaoo text generation
approach proves inadequate for such ultra-long- -oooiooo planning tasks (Ye et al. (2025) Effective solutions may require agents to adopt human- like hierarchical decomposition or symbolic planning techniques, executing subtasks to achieve final objectives through sequential decisioning.
Diversity and Openness of Travel Requirements. ChinaTraael surpasses TravelPlanner and Trip- Planning in diverse requirement modeling. As Fig. shown:::: (1) Constraint volume: Travel exhibits approximately Gaussian distribution (6-12 constraints per query) versus TravelPlanner's 's simplicity bias (55 constraints) and TripPlanning's ' limited diversity (allowing up to 16 constraints but spanning only two types). (2) Combinatorial capacity: TravelPlanner's atomic constraints yield only 10 combinations, while ChinaTTravel scales exponentially from 15 (synthetic) to 100 types (human1 1000 test), including 85 novel constraints formulated through Tab. [ss compositional system. We further investigate co-occurrence of constraint types within individual queries, we categorize basic concepts in our DSL into seven clusters as visualized in Fig. 4b In ChivaTraavel, the co-occurrence distribution follows Zipf's s law (Adamic & Huberman 2002) with a characteristic long-tu -til pattern, contrasting sharply with TravelPlanner (Fig. (Fig. whose synthetic data demonstrates relatively uniform frequencies. We could also find a strong correlation between cost-related con- straints and caunsortation/accommodaaa
requirements, which aligns with common sense given that transport and accommodation are primary cost components. These characteristics stem from systematic user studies that integrate the evolving, open-ended nature of travel requirements into our benchmark. Users continually introduce novel composite constraints, making it impossible to exhaustively enumerate all possibilities during development. By preserving scalable verifiability through our compositional DSL design, China can embrace an evolving requirement space, thereby systematically revealing and rigorously evaluating open-world challenges of travel planning.
Contextual Grounding for Implicit Intent. From human queries, we observe that travel intent is often expressed implicitly, leading to contextual ambiguity that is not directly aligned with predefined database attributes. For example, when users express intent for "local cuisine", which contextually maps to <Benbang cuisine> in Shanghai versus <Beijing cuisine>> in Beijing. Another representa- tive case involves users specifying "traveling with children who cannot eat spicy food", requiring agents to logically exclude Sichuan and Chongqing cuisines from restaurant selections, beacuse both of them are well-known as their spaicy style. These observations arise the necessity for travel agents to conduct contextual grounding that bridges arbitrary user expressions with verifiable symbolic semantics in databases, a critical challenge inadequately supported by existing benchmarks (Zheng et al.) 2024 Xie et al.) (024)). To systematically investigate this challenge, we designed aux- iliary task, Intent Grounding. It involves replacing all explicit POIs in DSL-defined constraints with <placeholde>>> tag, requiring LLMs to complete masked-DSL sentences through contextual grounding. This simplified formulation isolates POI inference from full DSL generation. We further categorize POIs as Literal (explicitly mentioned in user queries) Semantic (requiring cultural/con- textual inference). Quantitative analysis shows 78.4% of DSL statements from Human 1000 contain Semantic POIs needing contextual grounding, contrasting sharply with TraveIPlanner's 5.4% rate
5

---

Published as a conference paper at ICLR 2026
Traueffarner
Deeplack
Deeplelk
100
100
Liznd PSI Somards
Trai
LaarliN Somanac
Tall
(a) TravelPlanner
(b) ChinaTravel
(c) Intent Grounding
Figure 4: Co-occurrence distribution of differnt constraints on TravelPlaner (a) and China Travel's Human1000 (b). (c) The unsatisfactory performance of advanced LLMs on the auxiliary task, intent grounding, reveals the challenges of open contextual grounding in ChinaTravel dataset.
that predominantly requires literal mapping. Experimental results from DeepSeek-VV V3 and GPT-4o are shown in Fig. 4cc Both models achieve the accuracy over 90% on TravelPlanner, where semantic POIs follow simplistic synthetic patterns. However, on ChinaTravel'' authentic Semantic POIs, performance significantly declines (DS: 94% - 76%, GPT: 79% ~ 53%). This performance gap underscores the ChinaTrravel's contextual grounding challenges for real-world travel planning.
4 EMPIRICAL STUDY
recognized for their world-leading performance. We also examine the open-source LLMs, Sy QWen3-8BB t: Llama3. 1-8BB and | Mistrall77B selected based on their computationally efficient 7B/8B scales, which enables practical deployment in resource-constrained academic computing environments. We also report the additional results with large reasoning models, like DeepSeek-RI, at App. H.]
LLMs. We evaluate the state-of-the-art LLMs, @ DDeepeek-VV V3 OpenAI GPT-4o
Metrics. We examine the Delivery Rate (DR), Environmental Pass Rate (EPR), Logical Pass Rate (LPR), and Final Pass Rate (FPR) from (Xie et al., (024). For EPR and LPR, we report both Micro (the average proportion of satisfied constraints per plan, allowing partial credit) and Macro scores (the percentage of plans that strictly satisfy all constraints). The computation details of all metrics are provided in the App. G To address potential evaluation biases caused by unrealistic constraint prioritization, e.g., misreporting costs to satisfy budget requirement, we design novel metric, Conditional Logical Pass Rate (C-LPR). It assesses the success rate of travel plans that first satisfy environmental constraints before meeting logical requirements, thereby ensuring logical validity within realistic contextual boundaries. The introduction of C-LPR provides a more rigorous viewpoint for quantifying meaningful constraint satisfaction.
C-LPR- EpEp 1 passsed Enw.p) Eeec, passed((-r)
Eper Cpl
P is the plan set, Cp is the constraints set for plan p. and passed(c, p) indicates whether p satisfies c.
Methods. In this work, we mainly focus on the training-fiee free methods with both pure-LLM-bere based
and neuro-symbolic solutions. For the former category, we implement ReAct (Yoo et al. 2023b), 3b), a
reasoning-and-acting widely-adopted reasoning and-acting framework, along with its Act-only ablation variant. We exclude Reflexion (Shinn et al.) 2024) due to its performance being similar to ReAct on the TravelPlanner (Xie) al. 2024) and the high economic overhead associated with the larger input token size. For neuro-symbolic methods, we assess three frameworks: (1) TTG aJu et al., 2024), which converts natural language requirements into mixed-integer linear programming formulations for solver-based optimization. We adapt their formulation into ChinaTravel. The rapied growth of transformed constraints in TTG becomes computationally prohibitive. To address this, we employ L.LMs to extract the most relevant POIs for constraint reduction, with detailed linear constraint formulations and experimental configurations provided in App. K (2) LLM-modulo (Kambhampati et al. 2024 Gundawar et al.) 2024), employing ground-truth symbolic verification to guide iterative LLM self- refinement, which could be regrad as an enhanced variant of Reflexction. To ensure compatibility with mainstream LLMs, perform POI subsampling within 64K context window. (3) NeSy Planning, extending prior NeSy pipelines (Hao et al.) 2025 Pan et al. 2023 Yao et al.; 2023a Xiong et al. 2025 2026) through our DSL. enhancements to address complex multi-day, multi-POl itineraries.

---

Published as a conference paper at ICLR 2026
4.1 NEURO-SYMBORE PLANNING
Overy
Currest likeary Flan
Mtngghal and wookl
Tralk, GJM gli Hinglee Ralkeyy Siaken
Bejjnn days some munuere ent, and your and Tape to as maay atrictinn poodil Plans gire tand plus
This subsection presents a NeSy solution as a preliminary baseline for ChinaTravel This solu- tion consists of two stages. (1) NL2DSL trans- lation transforms natural language queries into logic and preference DSL requirements. We use Reflexion (Shinn et al., 2024) and DSL syntax checker to iteratively assist the LLMs (5 rounds in experiments). (II) Interactive search uses neuro-symin symbolic solver to sequentially ar- range activities, guided by symbolic sketch and LLM-driven POI recommendations, generating a multi-day itinerary with DSL validation. If constraints are violated, the process backtracks until a feasible solution is found. To ensure fair- ness, the symbolic sketch search is limited to 5 minutes per query, excluding L.MM inference time. To observe the performance across the two stages, also evaluated the planning results based on the Oracle DSI.. In App. [ we provide the search algorithm's 's pseudo-code and LLM prompts to enhance reproducibility and support future research.
06.2 - 13:12
Miero 1816 1856 walking flaer
LLM Extraction
Validated Plan
14:0 11.50 coe
Plansing, step stepe
2. A-rpe musesanoo
(i T Pakus Mtszzs
siatting
type
Beijjig
otrictions, restagaats hotel??
4 L.Monin fiel
total budget 5000
1533 1726 bik 17
13:33 -ioa acstanii bobare cinee.
Plan Verification
Friled
Campleted Plan
11 Si Nastu Casen:
Taxl use Restsnnn
bJ Armengg Jangite Units
Which POI Polshauld visi
DFS
Ride Faoe Aa
Typely Mistion Rejing Casiae
Sij Marki
knnvrrooamaa dack
Figure 5: NeSy Planning with search-based solver.
4.2
MAIN RESULTS
Table 2: Cost per query across different methods.
Based on the results presented in Tab. and we have the following finding and analyses:
Method
#Input #Output
sr() @(
Act
88K
2K
007
.219
Pure LLMs struggle in ChivaTravel. The DR evaluates the capability to generate valid ISON travel plans (see Fig. D. While high DR val- ues indicate that advanced LLMS can produce structurally correct outputs, the near-zero EPR reveals their fundamental limitations in acquiring and strictly adhering to required constraints. The sole exception is the DeepSeck, which achieves the 6% EPR and 5% FPR at easy level. A plausible explanation is broader training- -dta coverage for Chinese queries. ReAct (one-shot, GPT-4o) excels in Macro LPR but achieves no FPR, suggesting it circumvents constraints via shortcuts. Our C-LPR metric offers a more reliable measure of logical constraints, serving as a supplement to FPR. As shown in Tab. D purely neural baselines require large input/output tokens and correspondingly high cost. With GPT-4o, the average cost is $2.43 per query, yet they produce on constraint-sate sattsfii- plans. Given the substantial cost and their persistently low FPR, further pure-neural variants offer diminishing returns under our budget. We therefore concentrate on NeSy solutions.
ReAct (0-shot)
206K
3K
.021
.638
ReAct (1-shot)
1058K
4K
.081
2.43
LLM-modulo
362K
11K
.025
1.12
NeSy Planning
467K
3K
.003
.306
The Inadequacies of Existing NeSy Approaches. TTG's complexity grows rapidly with the size of the POI candidates and the temporal discretization: the number of constraints scales on the order of O(N3T) with N POIs and T time slots. Even after subsampling to (N = 22) and using 1-hour slots = 2-day instance contains the order of 600,000 constraints. We TTG with SCIP solver, allocated relaxed 15-minute search limitation per query. This configuration yielded only 18% valid solutions on easy-subet instances, with the FPR further reduced to 8% due the solver's pruning heuristics. Fig. 66 illustrrts the solve time of TTG on 1-3 day itinerary. Within the time limit, solutions were found for only 23% for 2-day and 6% for 3-day itineraries. LLM-lodulo introduces an oracle symbolic verifier to detect constraint violations and feeds back error messages to the LLM for iterative plan revision. Fig. (6b] depicts the error dynamics over 10 refinement rounds. GPT-4o attains the lowest cumulative error (/4 3.2 + 0.8), followed by DeepSeek (μ 5.1 + 1.2). However, their rectification capacity, quantified by successfully rectified errors per iteration, rapidly decays to < 1 after 3-5 rounds, indicating diminishing returns from further refinements. Smaller models (Qwen3-8B 8B and Llama3-8Bi show higher per-step rectification, but also introduce more emergent errors, yielding no significant refinement across iterations. Taken together, the perifier-feedbat loop, effective on earlier travel benchmarks, does not scale well to complex multi-day itineraries: after a few rounds, refinement stalls while additional iterations incur extra cost and latency.
7

---

Published as a conference paper at ICLR 2026
Table 3: Main results of different L.Mss and planning strategies on the ChinaTravel benchmark.
EPR
LPR
EPR
LPR
DR
C-LPR FPR
C-LPR FPR
Mic. Mac. Mic. Mac.
Mic. Mac. Mic. Mac.
Easy (#300)
Human- Val (#154)
70.4 49.9
0
64.6 30.6
0
Act
97.5 70.8
0
86.8
68.6
0
43.3
0
41.9
19.6
0
36.4
29.5
0.65
35.2
16.2
0.38
0
ReAct (2cro-shot)
98.4
0
71.3
33.0
0
96.1
50.5
72.4
32.5
0
0
67.5
6.00
74.1
52.3
5.77
5..2
55.2
57.3
2.59
64.6
44.2
1.71
2.59
ReAct (onc-shot)
98.2
0
89.4 70.6
0
69.5 46.3
0
63.6
46.8
0
0
75.3 75.3
70.4 52.6
70.4
52.6/51.9
53.2
52.5
47.0
37.6
46.5
37.0
NeSy Planning
75.0 64.0 73.5 63.3
61.7
60.6
45.4
50.1 45.4
40.9
29.8
38.5
27.9
72.3 34.0 70.4 49.6
32.6
28.3
47.4 42.2
36.2
27.2
34.4
25.3
32.0 31.3 29.1 21.0
28.3
21.0
25.8 24.0
22.3
12.3
20.5
11.0
M
30.3 30.3
30.3
27.6
19.6
27.6
39.6
38.2 37.6
32.7
18.8
32.2
18.8
TTG (oracle)
21.3 8.66
17.2 15.0
8.23
8.66/9.09
12.8 2.59 7.65
5.19
2.39
1.29
48.3 4.33 58.4 43.6
4.11
61.6
90.2 2.59 75.9 51.2
2.75
2.59
ALMMModulo+
91.6 88.2 7.66 95.5 84.6
7.66
7.00
91.5 87.2 3.24 92.9
66.2
2.87
3.24
(Oracle Verifier)
30.0 80.5 0.0 62.7 25.0
0.0
0.0
35.0 75.3 0.0 61.6 19.4
0.0
0.0
69.4 0.0 55.2 8.33
19.4 74.1
43.4 5.19
0.0
0.0
M
10.3 90.5 0.0 39.1 9.0
0.0
0.0
3.24
92.2
0.0 31.4 4.54
0.0
0.0
82.6 81.7 75.0 82.2 75.3
75.0
74.0
59.6
$7.7
53.8 46.1
52.0
45.4
NeSy Planning*
66.6 66.7 66.0 64.6 63.6
64.6
62.6 46.9 42.9 47.6 40.9
43.9
40.9
(Oracle Translation)
69.3 69.3 59.3 70.2 59.6
59.3
57.9 55.1 54.5 48.0 42.8
47.6
40.9
M
52.6 52.6 50.4 45.3
50.4
40.6
40.9
42.8 42.8 37.7 28.5
37.7
27.9
33.3 33.2 32.6 32.1 32.0
31.4
32.3
29.2 29.1 26.6 25.4 20.1
23.4
19.4
Human- Test (#1000)
NeSy Planning* (Omale Translation)
44.6 44.5 42.6 38.7 23.3
37.6
23.3 60.6 60.3 59.0 53.6 32.0 52.5
31.6
NeSy Planning
37.3 37.2 35.0 30.7 11.3
29.2
11.3 27.8 27.8 27.1 24.8 12.8
24.4
12.8
36.6 36.5 34.6 29.6 6.43
28.5
6..3
41.1
41.1
40.6
34.6
13.8
34.2
13.8
Nesy Planning provides promising solution. Our NeSy Planning method orchestrates tool use and planning via symbolic programs while utilizing LLMs to parse natural-language requirements and prioritize POIs. By decoupling understanding (flexible natural language handling), planning (DSL- guided backtracking vigricccii/eeiicattoo) and actioning (precise tool execution). it improves adaptability and adherence constraints context-rich long-horizon settings. Across the evaluated subsets, outperforms TTG and LLM-modulo, even without the help of oracle translation. With the DeepSeek as backend, it achieves FPRs of 52.6%, 37.0% and 23.3% three subsets, highlighting the effectiveness of NeSy solutions for travel planning with complex constraints. On the human-ve val and human-test subsets, these gains persist, suggesting robustness unseen constraint compositions.
Challenges Persist for Nesy Planning. The performance gap between standard and oracle modes underscores the importance of DSL translation in NeSy planning. Inadequate translations may result in plan searches failing to meet user requirements, while incorrect translations can misguide the search, making feasible solutions unattainable. We conclude with three challenges and provide the corresponding cases in the Fig. 9 (1) DSL Syntax Compliance: As evidenced in Fig. 7al while the reflexion process with syntactic checking effectively reduces parser-level errors, it inadvertently triggers constraint dropping across multiple LLMs. For Qwen3-8B, Llama3-8B, and Mistral-7B. the number of DSL constraint clauses decreases across iterations. Notably, GPT-4o generates approximately two fewer constraints than DeepSeek-Vi -3 on average under the same loop. Although
8

---

Published as a conference paper at ICLR 2026
Frequency
50
I-ay wip
day
Paniring Free
traggei heer
Taxl Depbost- Reckee
GF 4o Ractffl
Ukmo Roatted
Oweah Ructiled Marsh Heniie
30-
3-day
Tiite Out
10
10 Scarch Time (min)
10
Solve Time of TTG.
Refinement of LLM-modullo
Figure 6: (a) The high computational complexity of TTG renders it infeasible for real-world multi-day itineraries. (b) L.M- error correction declines during iteration, causing emergent errors.
Enne (1 Paweed 1881 Counit)
Matched Unseen Synlax
Matched Seen Syntax
Unmatched Unseen Syntax
Unmatched Syntax
9. 9.7%
0.8%
8.3%
1.2%
7. 13 1.0%
14 7%
15.1%
14.6%
74.8%
75.4%
77.3%
DespSeck-V3
GPT4c
Qwadd-9
Miarsi-1
DeepSeck-V3
GPT-4o
Qwen3-8B
(a) Syntax error of Iranslation.
(b) Syntax generalization DSL translation.
Figure 7: Challenges in NL2DSL translation.
this conservative strategy enables rapid error convergence (achieving zero detected errors within limited iterations), it risks oversimplifying constraint specifications, critical dependencies may be prematurely discarded, ultimately yielding solutions that fail to satisfy complex requirements. This conservative pattern often drives fast convergence to parser errors within few rounds, but may prune required constraints and under-specify the plan, leaving the outcome cannot satisfy complex queries. This observed conservatism on constraint extraction likely contributes GPT-4o's underperformance on Human-155 154 and Human- 1000 compared with DeepSeck-Vi V3. (2) Contextual Grounding: In the Sec. 3 we have provided a quantitative analysis for this challenge. Overcoming this might require domain-adaptive training, enabling LLMs to better interpret implicit user intent. (3) Unseen Concept Composition: Real-world requirements are diverse and open-ended, so unrealistic to expect models to encounter all possible needs during development. A more realistic way to emulate human reasoning by generalizing existing knowledge to novel problems. Fig, 7b compares three LLMs seen unseen DSL structures under POI-anonymized evaluation with syntax-level pattern matching. Unseen compositions constitute 84% of cases but achieve only 12% structure alignment (9% overall when weighted by frequency), whereas seen patterns (16% of cases) reach 93% accuracy. This gap holds across the evaluated LLMs, which perform well on seen patterns but drop sharply unseen concept compositions, suggesting limited compositional generalization.
In summary, NeSy methods outperform L..M-only baselines on constraint satisfaction, yet open-world challenges remain. With authentic queries and DSL-based compositional validation, ChinaTravel surfaces these limitations and delineates actionable directions for further research.
The substantial performance gap between standard NeSy planning and its oracle- DSL variant indicates that the primary bottleneck lies in constraint grounding, i.e., the ability to faithfully translate open-ended natural language into compositional constraints. General- curppoe-purpose LLMs that rely solely on in-context prompting still struggle to generalize to unseen constraint syntax (as detailed Fig. 77) and leave considerable room for improvement in open-ended semantic grounding (as detailed in Fig. 4cp. Although ChinaTravel provides (query, DSL) pairs for supervised fine-tuning and a sandbox with verifiable signals for reinforcement learning strategies, achieving robust compositional generalization remains open challenge, likely requiring sophisticated data
Path Forward:
sampling and augmentation strategies (Wu et al. 2023: Akyürek et al., 2020) as well as advanced learning paradigms (Yang et al. 2024 Liu et al. 2020: Lake 2019). Since this work primarily serves as a benchmark to identity the gap between current research and real-world scenarios, rather than to deliver complete solution, leave the development of such methods to future work.

---

Published as a conference paper at ICLR 2026
Nifly Plarning
B3Q
PE:
PD6
R1
Daily attractions
1.33
Tronsport time
27.0
Aggropainn Win
R3
PS
PO
Iransport time to rest.
222
26.8
22.8
Food oest rabio
0.19
0.29
0.32
6/8
610
640
Hotel cost. 1
1350
658
519
Distance POI
30.9
30.6
26.1
(a) Ablation on preference ranking.
(b) Parcto Win rates (PEQ v.s. BQ).
Figure 8: Empirical Analysis on Planning with Preferences.
4.3 ABLATION STUDY WITH PREFERENCE
Preference comparisons are meaningful only when both environmental and logical constraints are satisfied. Given the limited FPR achieved by existing methods, we perform a separate analysis of preference optimization here. Specifically, we sampl 50 queries from the casy subset that NeSy- DeepSeek-Oraz Oracle successfully passed seed samples. Based on these, two experimental settings were designed to explore the roles of LLMs and symbolic search.
Single-Preeerence Optimization. We first evaluated scenarios with single preference objective using six subsets created from user surveys. Three comparative scenarios were designed to explore the roles of L.MSs and symbolic search in optimizing preferences during NeSy Planning: (1) BQ: Baseline solutions without preference consideration. (2) PEQ: LLM-enhanced recommendations with natural language preferences. (3) PDS: Hybrid symbolic search optimizing preference objectives under 5-min constraints. From the results in Fig. 8a (where + / indicate waxiimzzatio//mniiizzttoon we cound find that: (1) PEQ outperforms BQ 5/6 preference scenarios, confirming LLMs' capacity to interpret natural language preferences during POI POI ranking. (2) PEQ underperforms on P2 (minimizing transport time to restaurants), likely from LLMs' misinterpretation of complex spatiotemporal constraints. These results support the DSL's scalability for preference optimization and highlight the need for more efficient algorithms for preference-aware planning.
Trade-off'. Real-world planning may involve balancing multiple, potentially conflicting preferences. To address this, we constructed 15 test subsets by pairing the six preferences (PO-P5) into all possible combinations (e.g.. "maximize daily attractions" + "minimize hotel cost"). Excluding PDS due to the complexity of symbolic weighting, we compare PEQ against BQ using Pairwise Pareto Win-Rate. PEQ is declared the winner (1.0) if it generates feasible plan that Pareto-dominates BQ (strictly better on at least one metric without degrading the other) and the loser (0.0) if BQ conversely dominates PEQ. Cases where neither dominates both fail constraints recorded as ties (0.5). Fig. 8b illluutttaees the win rates and their aggregation across 15 test settings. The results reveal meaningful structures in conflict resolution: PEQ performs well when jointly optimizing synergistic attributes, such as "maximize food cost ratio" + "minimize hotel cost" (P3 & P4, Win Rate 0.58), "less inner transports time" + "minimize average transport time to restaurants" (P1 & P2, Win Rate 0.45). In contrast, PEQ underperforms in some cost-sensitive combinations such as "minimize inner-city transport time" and "minimize hotel cost" (P1 & P4, Win Rate 0.1 These findings underscore the current limitations of LLMs in navigating rigid trade-offs between spatiotemporal efficiency and financial constraints, identifying a critical direction for future research.
5 CONCLUSION
We present ChinaTravel, an open-ended benchmark for multi-day multi-POI travel planning focused on authentic Chinese needs. It addresses gaps in prior benchmarks by pairing open-ended human queries with a DSL-based framework for compositional constraint validation, enabling evaluation of feasibility, constraint satisfaction, and preference comparison. The empirical analysis reveals the potential of aeuro-symboliy methods on constraint adherence. The open- world challenges identified, contextual grounding and compositional concept generalization, suggest actionable directions for future work. We hope ChinaTravel will facilitate progress in LLM- travel planning by providing standardized evaluation framework and highlighting key challenges for improvement.
10

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGMENT
This research was supported by Leading-edge Technology Program of Jiangsu Science Foundation (BK20232003), the Key Program of Jiangsu Science Foundation (BK20243012, BG2024036), Natural Science Foundation of China (62576162), and the Fundamental Research Funds for the Central Universities (0221 14380023).
ETHICS STATEMENT
We adhere to the ICLR Code of Ethics and conducted proactive review of data collection, curation, evaluation, and release.
Potential Positive impacts. ChinaTravel is a research benchmark for complex, real-world trip planning, by stressing compositional constraints and verifiable outcomes, it aims to catalyze more reliable, constraint-aware assistants and to facilitate cross cross-disciplinary research. Its positive societal impacts include: (1) Improved Travel Plamning Effectiveness: By rigorously testing agents' ability to handle multi-day itineraries and combinatorial constraints, this benchmark encourages the creation of more robust AI assistants, potentially reducing the time and effort users spend on organizing trips. (2) Validation for WorldWorld Applications: The benchmark establishes a critical foundation for deploying language agents in practical travel planning settings, where multi-objective planning and constraint satisfaction are essential. The release of this benchmark bridges cutting-edge LLMs with classical neuro-symbolic AI paradigms, fostering cross-disciplinary collaboration between academia and industry. It promotes the reliable, constraint- Al systems, while accelerating innovation in both foundational planning capabilities and real-world deployment scenarios.
Potential negative impacts. It largely depend on how future systems built upon this benchmark are deployed. For instance: (1) Bias and Fairness: agents inherit biases from training data or misalign with diverse user preferences, they might disproportionately recommend certain POIs services. Mitigation requires ongoing faimess audits and inclusive data practices. (2) Misuse Risks: Malicious actors could exploit highly capable planning agents generate deceptive itineraries manipulate travel services. Such risks underscore the need for ethical guidelines and safeguards in downstream applications. ChinaTravel is released for research purposes only. Any real-world deployment should include additional safety engineering, for example, explicitly warning users that agent-genn generated plans are suggestions, and implementing verification mechanisms (e.g.. feasibility and constraint checks) before adoption.
Language and Regional Scope (Bias Considerations). Our benchmark focuses on Chinese cities and collects requirements from native Chinese speakers because, for POI-rich, locale-specific travel planning, interacting in the target user's 's language yields more faithful intent capture and more coherent system behavior. This mirrors common practice in domain-specing systems (e.g., TravelPlanner (Xie et al.) 2024) uses English for U.S. scenarios). While our initial release centers on Chinese due realistic usage and practical constraints (API costs and compute/latency budgets), the core components are language- and
the tool- -ooongroddded sandbox, the DSL-based verification framework, and the identified open-wont world challenge are independent of any particular language. Future iterations will extend ChinaTravel'ss 's language coverage to address global tourism demands. Our goal is not to privilege one language or culture, but to start from high-fidelity setting where users naturally articulate open-ended, diverse travel requirements, enabling transparent, generalizable evaluation of reliable planning agents that better align with real-world deployment.
REPRODUCIBILITY STATEMENT
An anonymous, downloadable codebase and the dataset splits are provided in the supplementary attachments (with README that lists dependencies, exact run commands, and config files). The main paper and appendix together specify all components needed for reruns: benchmark design details and data details (App. D). tutorials for the DSL and preferences (App. D.3 and D.6). the search sketch, psdudo-code, and prompts for our NeSy Planning baseline (App. F evaluation protocol and metrics (App. Gp. We also document scientific artifacts (availability & licensing) in App.
11

---

Published as a conference paper at ICLR 2026
REFERENCES
Lada A Adamic and Bernardo A Huberman. Zipf"s law and the internet. Glottometrics, 3(1):143-150 2002.
Ekin Akyürek, Afra Feyza Akyürek, and Jacob Andreas. Learning recombine and resample data for compositional generalization. arXiv preprint arXiv:2010.03706, 2020.
Murray Campbell, A Joseph Hoane Jr, and Feng-hsiung Hsu. Deep blue. Artificial intelligence, 134 (1-2):57-83, -3,5778,,22 2002.
Aili Chen, Xuyang Ge, Ziquan Fu, Yanghua Xiao, and Jiangjie Chen. TravelAgent: An Al assistant for personalized travel planning. arXiv preprint arXiv:2409.08063 2024.
Jiefeng Chen, Jie Ren, Xinyun Chen, Chengrun Yang, Ruoxi Sun, Jinsung Yoon, and Serean Ô Ankk. Sets: Leveraging self-verification and self-correction for improved test-time scaling, arXiv preprint arXiv:2501.19306, 2025.
Jihye Choi, Jinsung Yoon, Jiefeng Chen, Somesh Jha, and Tomas Pfister. Atlas: Constraints-aware multi-agent collaboration for real-world travel planning. arXiv preprint arXiv:2509.25586. 2025.
Wang-Zhou Dai, Qiu-Ling Xu, Yang Yu, and Zhi-Hua Zhou. Bridging machine learning and logical
reasoning by abductive learning. Advances in Neural Information Processing Systems, pp.
2811-2822, 2019.
Shujie Deng, Honghua Dong, and Xujie Si. Enhancing and evaluating logical reasoning abilities of large language models. In Proceedings of the ICLR 2024 Workshop Secure and Trustworthy Large Language Models, 2024.
Jerry Fodor. The language of thought. Harvard University Press, 1975.
Jerry A Fodor. LOT The language of thought revisited. Oup Oxford, 2008.
Ambros Gleixner, Gregor Hendel, Gerald Gamrath, Tobias Achterberg, Michael Bastubbe, Timo Berthold, Philipp Christophel, Kati Jarck, Thorsten Koch, Jeff Linderoth, et al. Miplib 2017: data- driven compilation of the 6th mixed-integer programming library. Mathematical Programming Computation, 13(3):44344999 2021.
Atharva Gundawar, Mudit Verma, Lin Guan, Karthik Valmeekam, Siddhant Bhambri, and Subbarao Kambhampati. Robust planning with Ilm-modulo framework: Case study in travel planning. arXiv preprint PxXi::2405.20625 2024.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14953,1496,, 2023.
Sajal Halder, Kwan Hui Lim, Jeffrey Chan, and Xiuzhen Zhang. A survey on personalized itinerary recommendation: From optimisation deep learning. Applied Soft Computing, 152: 12000 2024.
Yilun Hao, Yongchao Chen, Yang Zhang, and Chuchu Fan. Large language models can solve real- world planning rigorously with formal verification tools. In Proceedings of the 2025 Conference of the Nations the Americas Chapter of the Association for Computational Linguistics, Albuquerque, New Mexico, 2025.
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? Proceedings of the 12th International Conference on Learning Representattinns, 2024.
Da Ju, Song Jiang, Andrew Cohen, Aaron Foss, Sasha Mitts, Arman Zharmagambetov, Brandon Amos, Xian Li, Justine Kao, Maryam Pazel-Zarandi et al. To the globe (ttg): Towards language- driven guaranteed travel planning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 240-249, 2024.
12

---

Published as a conference paper at ICLR 2026
Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy. Position: Lims can't 't plan, but can help planning in Ilm-modulo frameworks. In Forty-first International Conference Machine Learning, Vienna, Austria, 2024.
Brenden M Lake. Compositional generalization through meta sequence-to-sequenae learning. Ad- vances in neural information processing systems, 32, 2019.
Qian Liu, Shengnan An, Jian-Guang Lou, Bei Chen, Zeqi Lin, Yan Gao, Bin Zhou, Nanning Zheng, and Dongmei Zhang. Compositional generalization by learning analytical expressions. Advances in Neural Information Processing Systems, 33:1 1416-11166611 11427, 2020.
Weiyu Liu, Geng Chen, Joy Hsu, Jiayuan Mao, and Jiajun Wu. Learning planning abstractions from language. In Proceedings of the 12th International Conference on Learning Representations, 2024.
Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Lue De Raedt. Deepproblog: Neural probabilistic logic programming. In Advances in Neural Information Processing Systems, pp. 3753-3763, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing Atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning. In Findings of the Association for Computational Linguistics: EMNLP, pp. 3806-3824, 2023.
Steven T Piantadosi, Joshua B Tenenbaum, and Noah D Goodman. The logical primitives of thought: Empirical foundations for compositional cognitive models. Psychological review, 123(4):392, 2016.
Vibhor Sharma, Monika Goyal, and Drishti Malik. An intelligent behaviour shown by chatbot system. International Journal of New Technology and Research, 3(4):263312, 2017.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, 2024.
David Silver, Julian Schritwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy P. Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature, (5507767)):354-33) 2017.
Yihong Tang, Zhaokai Wang, Ao Qu, Yihao Yan, Kebing Hou, Dingyi Zhuang, Xiaotong Guo, Jinhua Zhao, Zhan Zhao, and Wei Ma. Synergizing spatial optimization with large language models for open-domain urban itinerary planning. CoRR, abs/2402.07204,, 2024.
Po-Wei Wang, Priya L. Donti, Bryan Wilder, and J. Zico Kolter. SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver. Proceedings of the 36th International Conference Machine Learning, pp. 6545-6554, 2019.
Xindi Wu, Hee Seung Hwang, Polina Kirichenko, and Olga Russakovsky, Compact: Compositional atomic-to-complex visual capability tuning. arXiv preprint arXiv:2504.21885 2025.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang. Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin. Shihan Dou, Rongx- iang Weng, Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huang, and Tao Gui. The rise and potential of large language model based agents: A survey. CoRR, abs/2309.00 00844 2023.
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for real- rorldworld planning with language agents. In Proceedings of the s1 International Conference on Machine Learning, 2024.
13

---

Published as a conference paper at ICLR 2026
Siheng Xiong, Ali Payani, Yuan Yang. and Faramarz Fekri. Deliberate reasoning in language models as structure-awart planning with an accurate world model. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 31900-31931, Vienna, Austria, 2025.
Siheng Xiong, Ali Payani, and Faramarz Fekri. Enhancing language model reasoning with structured multi-level modeling. In The 14th International Conference on Learning Representations, 2026.
Haoran Yang, Hongyuan Lu, Wai Lam, and Deng Cai. Exploring compositional generalization of large language models. In Proceedings of the 2024 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies ( (Volume Student Research Workshop), pp. 16 24, 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In Proceedings of the 11th International Conference on Learning Representations, 2023b.
Xi Ye, Fangcong Yin, Yinghui He, Joie Zhang, Howard Yen, Tianyu Gao, Greg Durrett, and Danqi Chen. Longproc: Benchmarking long-context language models on long procedural generation. arXiv preprint arXiv:2501.0544 2025.
Hongbo Zhang. Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Guiming Chen, Jianquan Li, Xiangbo Wu, Zhiyi Zhang, Qingying Xiao, Xiang Wan, Benyou Wang, and Haizhou Li. Huatuogpt, towards taming language model to be doctor. In Findings of the Association for Computational Linguistics: EMNLP, pp. 10859-10885, 2023.
Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Tze Cheng, Quoe V Le, Ed H Chi, al. Natural plan: Benchmarking ilms natural language planning. arXiv preprint arXiv:2406.04455 2024.
14

---

Published as a conference paper at ICLR 2026
TABLE OF CONTENTS
Introduction
2
China Travel Benchmark
2.1
Environment Information
2.2
Logical Constraint
2.3
Preference Requirement
2,4
Benchmark Construction
Benchmark Characteristics
Empirical Study
4.1
Neuro-Symbolic Planning
Main Results
4.3
Ablation Study with Preference
10
5
Conclusion
10
Acknowledgment
Ethics statement
11
8
Reproducibility Statement
11
The Use of Large Language Models
16
Discussion of Limitations
16
Discussion with Related Work
17
Detailed Design of ChinaTravel
17
DJ I Sandbox Information
17
D.2 City-wise Distribution Statisties
18
D.3 Tutorial of DSL Expression
18
D.4 Query Synthesis...
22
D.5
Diversity of Synthetic Data and Bias Mitigation
22
D.6
DSL Expression Preference
27
D.7
Benchmark Difficulty and Applicability
29
D.8 DSL Extension
29
Discussion with TravelPlanner
31
NeSy Planning
32
15

---

Published as a conference paper at ICLR 2026
Evaluation Metric in Competition
33
Additional Experimental Results
34
H.1
Results with Large Reasoning Model
34
Results on Medium Set
H.
Multi-Preference Comparison
................................
35
Analysis of Pure-LLM Methods
36
H.5
The Ablation Study of The Proposed NeSy Planning
37
H.6
Results on English Setting
37
Analysis of the Challenge Composition Complexity
38
Statements about Scientific Artifacts
38
Statements about Human Participants
38
J.1
Instructions Given To Participants
39
U.2
Recruiment And Payment
J.3
Data Consent
39
J.4
Institute Ethics Approval and Risk Mitigation
39
J.5
Risk Statement for Participants
40
J.6
Characteristics of Annotators
40
J.7
DSL Annotation for Human Data
40
J.8
Temporal Coverage of Human Data
41
K
The Implementation of TTG Baseline
41
K.
Constraints Formulation
K.
Experimental Setting
41
A
THE USE OF LARGE LANGUAGE MODELS
Polish Writing: used off-the-shelf LLMs as general- assist tools for sentence polishing during manuseript revision. All LLM M-assiisted edits reviewed and revised by the authors. LLMs are not eligible for authorship: the authors take full responsibility for all content.
Data synthesis: We LLM generate datasets easy and medium the components of the China'rravel becnhamrk. The complete procedure for synthetic data generation, including prompts, sampling settings, filtering, and human verification criteria, provided in the App. D.4
LLMs did not contribute to research ideation, experimental design, or core method development.
B DISCUSSION OF LIMITATIONS
Our research represents a significant step forward in evaluating the travel planning capabilities of language agents, but it is not without challenges. One limitation lies in its focus on Chinese travel planning. Due to the inherent differences in natural language, the translated versions of queries may fail to fully capture the challenges of understanding requirements in Chinese queries, potentially limiting its applicability global context. However, given the substantial demand within China's travel market, believe benchmark tailored to Chinese travel planning is both necessary and
16

---

Published as a conference paper at ICLR 2026
socially valuable. Although our benchmark is comprehensive, it may not encompass the full range of requirements encountered in real-world scenarios. The high cost of collecting authentic data has limited the number of human queries in our study. To address this, future work will focus on combining L.Mss with real user queries to automate the generation of a wider variety of human-like queries. Continuous refinement and expansion of our benchmark are crucial for more accurately reflecting the realistic travel planning needs.
ChinaTravel provides a verifiable, tool-equipped sandbox, but we currently focus on evaluation of prompt-based based methods and do not train tool-using agents with RL post-training. We defer these due to resource constraints (compute for large-scale interaction), and open challenges in trajectory synthesis (coverage and off-policy bias). Thus, we plan to explore tool-use trajectory synthesis and corresponding agent training future work.
C DISCUSSION WITH RELATED WORK
LLM-based Agents have demonstrated significant capability in understanding complex instructions and employing domain- specific tools to complete tasks, showcasing their potential in fields such as visual reasoning (Gupta & Kembhavi 2023), healthcare (Zhang et al. (023) and robotics (Liu) (et al. 2024). This reduces the reliance previous agents on domain-specific efforts, that is, either mainly following dpmein-specific rules to plan (rule-based agents, such as DeepBlue (Campbell] cct al. 2002) and Eliza (Sharma et al., 2017 ) or mainly learning from domain-speciffi data to plan based agents, such AlphaGo (Silver et al. 2017) and Atari DQN (Miih et al. 2013)). While the language agents have shown promising results in some domains, most of their planning scenarios are limited to simple tasks with single objective function and fail in the travel planning benchmark with complex logical constraints.
(reinforcement-leannie
Neuro-Symbolic Learning explores combine traditional symbolic reasoning with learning to enhance the reliability (Manhaeve et al., 2018 Wang et al.) 2019 Dai et al. 2019). In the era of large language models. Pan et al. (2023) presents the LogieLM integrates LLMs with separate symbolic solvers for various logical reasoning tasks. They first utilize LLMs to translate a natural language problem into symbolic formulation. Afterward, deterministic symbolic solver performs inference on the formulated problem to ensure the correctness of the results. Deng et al. (2024) supplement LogicLM with a Self-Reeinement Module to enhance the reliability of LLM translation. In the travel planning domain, Hao et al. (2025) presents framework with a similar pipeline. It first extracts the logical constraints from natural language queries and then formalizes them into SMT code. Thanks to SMT solvers being sound and complete, this neuro-symbolic solution guarantees the generated plans are correct and has basically solved the TravelPlanner benchmark with a 97% pass rate.
Travel Planning is time-consuming task even for humans, encompassing travel-related information gathering, POI selection, route mapping, and customization to meet diverse user needs (Halder et al. 2024 Natural languages one of the most common ways for users express their travel requirements. However, the ambiguity and complexity of travel requirements make it still challenging for LLMs to generate accurate and reliable travel plans. Xie al. (2024) presents the TravelPlanner benchmark for cross-city travel planning and reveals the inadequacies of Mine-LLM-driveer agents. TraveIPlanner generates user queries through LLMs and provides rigorous evaluation mechanism to verify whether the provided plans can meet the logical constraints in the queries. It has become a pivotal benchmark for language agents in real-world travel planning. Tang et al. (2024) study the open-domain urban itinerary planning where single-day multi-POI plan is required. They integrates spatial optimization with large language models and present system ITTNERA, to provide customized urban itineraries based on user needs. A concurrent work, TravelAgent (Chen et al. 2024), also considers a multi-day multi-POI travel planning problem for the specified city. It constructs an -Lwe-d system to provide personalized plans. However, due to the high cost of collecting and annotating real travel needs, they evaluate the proposed TravelAgent in only 20 queries. This also demonstrates the necessity of introducing new benchmark for travel planning.
17

---

Published as a conference paper at ICLR 2026
D DETAILED DESIGN OF CHINATRAVEL
D.1 SANDBOX INFORMATION
We started collecting travel information with the motivation of planning a multi-day, multi-POI itinerary in four aspects: attractions, accommodation, activities, and transportation. Developers first determine the POI description information that needs to be obtained from the user's perspective, such cuisine and hotel features. Based this feature set, collect public information construct the database. For the design of APIs, we directly support queries based on the regular expressions from LLM agents. At the same time, we expect the design of APIs to have similar features and characteristics to existing commercial APIs, enabling our dataset to be applicable to more realistic scenarios. The information our database contains shown in Table 4 and the APIs we offer is in we provide the detailed information of environment constraints in ChinaTravel.
Table 6
In Table 7
Tool
Information
Attractions
Name, Type, Latitude, Longitude, Opentime, Endtime, Price, Recommendmintime, Recommendmaatine
Accommodations
Name, Name Featurehoteltype. Latitude, Longitude, Price, Numbed
Restaurants
Name, Latitude, Longitude, Price, Cuisinetype, Opentime, Endtime, Recommendedfoood
Transportation
Transportation in specific city including walk, metro and taxi
IntercityTransport
Flight: FlightID, From, To, BeginTime, EndTime, Duration, Cost Train: TrainID, TrainType, From, To, Time, EndTime, Duration, Cost
Poi
Names of POIs(including intercity transportation hub) and their coordinates
Table 4: Sandbox Information
D.2 CITY-WISE DISTRIBUTION STATISTICS
Our POI collection was conducted on per-city basis, ensuring comparable distribution scales across urban datasets. Human queries exhibit a long-tailed distribution across cities, reflecting real-world travel patterns and highlighting practical deployment challenges for travel planning system. The detailed sandbox and dataset statistics are provided in Table
Table City-wise Statistics of Sandbox and Dataset
City
Attractions
Hotels Restaurants
Queries (Total)
Queries (Easy)
Queries (Human- Val)
Queries
Hluman-Testl
Boijing
334
400
469
210
30
28
152
Chengdu
332
378
229
Chongqing
346
372
436
191
36
16
139
Guangzhou
338
399
466
90
24
14
52
Hangzhou
376
377
457
195
33
10
152
Nanjing
322
372
123
18
Shanghai
359
402
483
180
37
25
118
Shenzhen
305
497
477
81
35
7
39
Suzhou
358
292
468
69
12
48
Wuhan
333
367
456
86
30
47
D.3 TUTORIAL OF DSL EXPRESSION
Here is a tutorial, that provides a step-by-step guide to utilizing ChinaTravel's Domain- Domain-Specific Language (DSL.) with predefined concept functions for expressing logical constraints and preferences.
18

---

Published as a conference paper at ICLR 2026
Tool
API
Does
Attractions
durractions_kes(cctty
Return list of (key, type) pairs of the attractions data.
attractions seliriottsssllltttttt...... key, func)
Return a DataFrame with data filtered by the specified key with the specified function.
attractions id is open(city, time)
id,
Return whether the attraction with the specified ID open at the specified time.
attractions cetitiioonn.narbyyciiiit..... point, topk, dist) attractions types
Return the top K attractions within the specified distance of the location.
Return a list of unique attraction types.
Accommodations
Return a list of (key, type) pairs of the accommodations data.
cttpoooddatonnssllettiiiii key, fune)
Return a DataFrame with data filtered by the specified key with the specified function.
Return the top K accommodations within the specified distance of the loca- tion.
point, topk, dist)
Restaurants
restaurants kestsurantskees((itt))
Return a list of (key, type) pairs of the restaurants data.
restaurants sel.eutccseeeecttttyy... key, func)
Return a DataFrame with data filtered by the specified key with the specified function.
cestaurants.id.ss.oop time)
id,
Return whether the restaurant with the specified ID is open at the specified time.
restaurants, seterratttnnnrbyccttty.. point, topk, dist) restaurants. wwwh bethhrecommended.fe food (city, food) restaurants cuuisunnttttcuuiiiectttt)
Return the top K restaurants within the specified distance of the location. Return all restaurants with the specified food in their recommended dishes. Return list of unique restaurant cuisines.
Transportation
goto(city, start, end, start time, transport type)
Return a list of transportation options between two locations with the specified departure time and transportation mode.
IntercityTrrnnspo
srapeiyy.taanspottseeecttstiiiie end_ch intercity .tpe, earli- est let_leave time)
Return the intercity transportation infor- mation between two cities.
Others
notedownn(description,, content)
Write the specified content to the note- book
plan(query)
Generates plan based on the notebook
content and query and report the plan done.
next. next_pagel page()
Get the next page of the latest Result his- tory if it exists. Because of the length limited, all returned DataFrame infor- mation is split into 10 rows per page.
Table APIs
19

---

Published as a conference paper at ICLR 2026
Category
Environment Constraints
Semantics
Intercity transportation events must oc- cur.
The first event and last event must be cross-city transports.
Cross-city
Transportation
Available Trains or Airplanes across cities.
The provided TrainID/FlighlDD, origin and destination should be valid in the travel sandbox.
Correct information of price, duration.
The price and duration information should match the travel sandbox. Provide number of tickets and cost of each inter-city activity. cost price tickets
Detailed cost on inter-city transporta- tion
Available Metro, Taxi Walking be- tween different positions. Correct information of price, distance, and duration.
The provided routes should be valid in the travel sandbox.
Inner-city Transportation
These details should match the travel sandbox.
Detailed cost on inner-city transporta- tion
Provide number of tickets/cars and cost. Taxi: people per car. cost = price x tickets, cost price
Available attractions in the target city
The provided attractions should be valid in the travel sandbox.
Attractions
Visiting during opening hours.
Activities must respect the attraction's opening time.
Correct price information. Detailed cost of attraction activity.
Must match the sandbox. Provide ticket number and total cost. cost = price x tickets
No repeated attractions.
Attractions should repeat across the trip.
Available restaurants in the target city Visiting during opening hours. Correct price information.
Must be valid in the travel sandbox. Same as above.
Must match the sandbox.
Restaurants
Detailed cost of restaurant activity. No repeated restaurants.
cost = price x tickets
Same restaurant should not be visited more than once.
Meals served in proper time slots.
Breakfast: 06:00-09:00;
Lunch:
11:00-4::00; Dinner: 17:00-20:00.
Available accommodations in target city.
Must be valid in the travel sandbox.
Accommodation
Correct price and room type.
Must match the sandbox.
Detailed accommodation cost. Required for trips over one day.
cost = price x rooms
A hotel necessary for multi-day trips.
Activity duration details.
Must include start and end time; end time must be after start.
Time
Activities in chronological order.
Events listed in order, respecting preced- ing transport arrivals.
Space
Transport info for changing positions.
If positions differ, the transport route must be included.
Table 7: Environment Constraints and Semantics in ChinaTravel Environment
20

---

Published as a conference paper at ICLR 2026
Logical Constraint
Transportation
The required type of transportation.
Attraction
The required type specified attractions.
Restaurant
The required type or specified restruants.
Accommodation
The number of rooms and the room type must meet the requirements.
The required features or specified hotels.
Budget
The total cost is within required budget.
Unseen Logical Constraints in Human data
POIs Time
The negation/conjunction hitiooonijinciioooiiiinnnniiiiii of given POIs
The duration of specific activities is within the limitation
Budget
The cost of specific activities is within the required budget
Table 8: Descriptions of Logical Constraints for two benchmarks. Constraints in black are common in both TravelPlanner and ChinaTravel. Metrics in brown are the metrics only in our benchmark.
Preference Requirements
Daily attractions 1
Visit as many attractions possible Minimize the travel time between POIs Minimize the travel time to restaurants Maximize the proportion of dining expenses Minimize accommodation costs Visit places as close to {POI} as possible
Transport time
Transport time to the restaurants | Food cost ratio f
Hotel cost
Distance to POI
Table 9: Descriptions of Preference Requirements in ChinaTravel benchmark.
DSL Overview In the main body of this paper, we have detailed the basics of our DSL in the Table The DSL a Python-like language designed to formalize travel planning requirements into executable code. It enables automated validation of itineraries against user constraints and preferences. Key components include: 1) Concept Functions: Predefined functions (c.g., activity poi_distance) that extract attributes from travel plans. 2) Operators: Logical (and, or, not), arithmetic (+, -, /). and comparison operators (<, >, =)) 3) Control Structures: Loops (for), conditionals (if), and set
operations (union, intersection). More examples provided in Figure 10
Syntax Compliance
Open Language Reasoning
Concept Composition
Query: Tour of would like to visit Shanghai for 2 days, tr local fist food, and strall along the Bund. Please powide with mavel plan.
go a.5. day trip Bejing friend We plan unved by hightspoal tain both ways and hope try some local specially frods.
Query: caneenlly Nanjing
Query:
tavelin alone Soon Nuijing Slonghui in muming for a day trip. plus
a. university campus and retum in the evening. making sure de easch the train back before
Transtation (-wen2.5-TBi:
DSL Translation (GPT-4o):
(actisty postontocctttyyy specified
type
(GPT-44):
uxtivity allacti-iikssiaan:::
typepetivity ['heasifiod dinner"
allactistiicgpllnt accivity timetccvityy 19900 roallt
Erron:
resturant type, restauuet typetactiiity targtt cityplas3)
name bactvvity definell
involid syntax (Catting-)
resalim(() (83E starent peti
Query: Curmant location: Changehoon I want to go Shonzhen zloze for a day, with budget of
Quers: My paronts and plus five-day taael fion Nasjing to Beijin watch the flag-nising commony. and we want stay at hotel near Tlananmen Square
Query: My bodlerr and ae planting to tovel from Stanghur Chimpring for 4 doys Apurt from the rosad-erip high-speed trannfight, we aim to spend more thas 3400 yuan Chosgeing
1000 RMB Please provide with trovel plan.
DSL. Transtarion (Qwnn2.-188j
resilt True
DSL. Transtation (GPT-4o):
aclivity allectisttice(piann
hotel sct set)
DSL Trancbation (GPT-44): costrd
factviity type activity) in (tmini brrpland':
allectestnsspiii
liternty Imiigort i set addiciisss typeattistyl) (inerli pransport set []]
(facthlyy aptlaitivlty) "acoommddatio::
allactivitiespplant
salipacivity pusiiiertaaiii
mustivity cosstactiityl iimonity. imnepot, costaactivty masportsscctittt resute-foudd coste:1400)
Krree:
resule (LBC
name Siterrity transport sot is noe definad
bosel names sell
Figure 9: Challenges in the Neuro-Symbolic Planning.
Core Concept Functions
We have defined 35 concept functions. Their definition and implementa- Below are common use cases:
tion is in Table 10, I1 12
and 3
21

---

Published as a conference paper at ICLR 2026
Example: Budget Constraint User Query: "Total expenses must not exceed 5000 CNY."
total_com cost 0
for act in all aattiiiiiiessee (plan)
total_cost += activity_cont cosyttcotttoooe (act)
total_cost += return total_cost <= 5000
(ectivity_trangpotte (act))
The function all activities(plan) iterates through all activities in the itinerary. The function activ- ity_ ity_cost retrieves the cost of each activity. The function innercity_transport pespporttcstttttt sums transportation expenses. Based on Python syntax, combining these concept functions can calculate the cost of the entire plan, thereby determining whether the budget constraints are met.
Debugging Tips (1) Syntax Validation: Use the python compiler to check for syntax errors (e.g.- missing colons, undefined variables). (2) Unit Testing: Test individual concept functions (e.g.. poi.distance) with mock itineraries. (3) Iterative Refinement: For ambiguous requirements (c.g..
"local cuisine"), map natural language to precise DSL concepts from sandbox information (e.g., restaurant tappeac,, city) = "Beijing Cuisine")..
Integration with Neuro-Symbolic Agents. (1) NL2DSL Translation: Convert user queries into DSL using LLMs (c.g., "Try local food" restaurant_yypc(POL, city) == "Beijing Cuisine" when the destination city is Beijing). (2) Symbolic Validation: Execute DSL code to verify plans against logical constraints. (3) Search Optimization: Use DSL-defined preferences (e.g., minimize(transport time)) to rank candidate itineraries.
D.4 QUERY SYNTHESIS
We designed common travel information (origin, destination, days, number of people) and logical constraints based on the nature of travel tasks. To facilitate scalable queries for ChinaTravel,, we randomly constructed query skeletons from the aforementioned information and used advanced LLMs to generate natural language queries from these skeletons. In practice, we provide the LLMs with more intuitive hard logic constraints ensure the LLMs do not make mistakes and use Python script to convert it after generating the query. The automatically generated data is categorized into two difficulty levels: In the Easy level, user inputs encompass single logical requirement, sourced from categories such as transportation, restaurants, attractions, and accommodations. In the Medium level, user inputs involve to logical requirements, introducing more complex constraints. During the generation, encourage the LLMs provide varied and human-liks expressions, necessitating deeper understanding and processing to accurately interpret and fulfill the user's needs. For instance, the logical requirement "taste Beijing cuisine" could correspond the natural language query: "Try local food in Beijing." We utilize prompt engineering to guide LLMs in refining natural language expressions facilitate automated generation. One of the prompts shown in Figure [I] Several examples of generated data is in Figure 12 As a result, we obtain the synthetic queries across diverse travel requirements, including 28 restaurant types, 23 attraction types, 29 hotel features, and more than 130 specific POIs
D.5 DIVERSITY OF SYNTHETIC DATA AND BIAS MITIGATION
This subsection provides detailed analysis of ChinaTravells hybrid query design, addressing concerns about synthetic data limitations and real-worke representativeness. When synthesizing data, we randomly constructed constraints based on the types and specific visit requirements of POIs such restaurants, accommodations transports, and attractions, thereby ensuring the diversity of the dataset. The synthetic queries are generated through L.LM-base paraphrasing techniques and systematically categorized into two tiers: Easy-tier queries contain single logical constraints (c.g., specific cuisine requirements), while Medium-tier queries combine 3-5 interdependent constraints (e.g., compound conditions like "budget < 3000 CNY + train transport hotpot dining").
To mitigate synthetic data bias and enhance diversity, two strategies were implemented. First, constraint combinations were deliberately diversified across temporal, spatial, and cost dimensions, as detailed in Table Second, a human validation layer filters out unrealistic --M-eerrrtted queries, such physically implausible itineraries like "visiting 10 attractions within one day."
22

---

Published as a conference paper at ICLR 2026
Function Name
Meaning
Implementation
def day_count (plan):
day_count
total days the plan
return (plan "itinerary"!)
people_count
number of peo- ple in the trip
def people_count (plan)
return plan[ "ppcpppleemmmbbrrr
start.city
start city of the plan
def start (plan)
return plan|"staaaei ("start_citty))
larget_city
larget of the plan
def (plaet)))
return plan "target_city"))
def
al (pla activity_list.
for day activiiy in plan Mitinerary");; for act in day_ say_acttvity (ortty"""ac""" ("activities)) activity list. abstad (act) return activity_lic list
allactivities
all the activities in the plan
def al
(plan):
count
for day_ activity in p "itinerarry" count +=
allactivities.. count
the number of activities in the plan
1en (day activiiy (naeeviitty"attvvities" ("activities")
return count
def dayactivities day): Sitivity_list for act plan "itinerrrr))r
dayactivities
all the activitics in the specific day [1, 2, 3,
Iday - 1 " activity_list. appppe (act) return activity_list
activity.cost
the of def activity_cost (activity): specific activity without trams- port cost
return activity, get ge ("cost", 0)
def pnaitty_positionaatttiie return activity. ("position", "")
activity-posi- tion
the position name of spe- cific activity
def sttisity_pricee(atttttttt return activity. get.:/ottttttrtaee ("price",
activity-price
price spe- cific activity
activity.tye type
the type of spe- cific activity
def return
"")
activity.
the number of tickets needed for specific ac- tivity
def activity_tickets (activityy)
return activity. get (eteee""""ie ("tickets", 0)
def activity srapssp
activity trans- ports
the transport information of
return activity. get (stiiitty.eeettraspororrr. ("transports", (])
specific activity
activity-- star
the start time of specific activity
def wwr.yyssttrr.ccieeeaaatiiii return activity.get
def activiity_en__iime (activiity return activity. get ("end_time")
activity.-
end time specific activity
end_time
Table 10: Concept Function
23

---

Published as a conference paper at ICLR 2026
Function Name
Mcaning
Implementation
def activity_time (activity):
start_time = activity. get ("start_time")
end_time activity get ("end_time")
start_time and end_time:
st_h, st_m
activity.time
duration
map (int, start_time.. (tt.:ttimme.ll((""))))
specific activity
ed_h, ed_m
map (int, end split
return (ed m mt + (ed_h h st h) 60
return -1
def ges_reccomend_taiee te poi):
select Attractions ().eleec selecc attration_infa
select (city, key="name",
poi recom- mend time
the recommend time of specific
func=lambda x: x == poi) .iloc [0]
recommend time (attreticn_infol (etecciio_iifffrrccceeeeeeeee "recommendmintiine") \
poi(attraction) in the city
60
return recommend_time
def poi (cit,,,ttnneei,ty, poil, poi2)
"ttart_tim==OO0
poi_distance
the distance be- tween POIs city
transport_type-"walll
goto Transpertation goto
return goto poil, poi2, start_time,
cransport_typer [0] [0] ("distance")
def Lnnereity.transpottccom (transports, mode)
cost for transpert in transports: if node is None or transport. ("type") node: cost += transport. ("cost", 0)
innercity-- transport_cost
the total cost of specific
innercity trans-
port
return cost
def pereriitytranspott__rrr (transportss)
price = for transport in transports: price += transport (prriee)
innercity-- transport price
price innercity trans- port
return price
def innercity_ttansport distancce (transports, mode=None):
distance =
for transpert in transports: if mode is None or transport.get ("typee") mode:
innercity
the distance of innercity trans- port
transport
distance
distance += transport.get (*di stance", 0) return distance
def def aalc_time_deltaaannddd start_time)
hourl, minul int inndd___mmmmi_lliii
[0]),
innercity
the duration of innercity trans- port
int
(1])
hour2, minu2 - int (start ilstart_timeettt time spl"
trame
int (start (start__timee.e.... (" (":")
return (hourl hour2) 60%
+ (minul minu2)
Table 11: Concept Function
24

---

Published as a conference paper at ICLR 2026
Function Name
Meaning
Implementation
def metro_tickets (transports);
metro_tickets
the number of metro tickets
return transperts [1] ["tickets"]
the type transport is metro
the number of def taxi_cars (transports)
taxi_cars
taxi if the type of trans- port is taxi
return transperts [0] ["cars"]
the number of def room_count (activity):
room_count
return activity get ("rooms", 0)
rooms of
commodation
the number of def room_count (activity): rooms of ac- commodation
room_count
return activity. get ("rooms",
room_type
the type of
def room_type (activity)
return activity, get ("roon_type", 0)
room of accom- modation
def restaurant_type (activity, target_city);
restaurants Restaurants
saleat_ffoo_ttee
restaurants.select.
target_city, key="name",
restaurant type
the type of restaurant's cuisine in the target city
func=lambda [Mcuisine"]
activity ("prityyppostiion)
if not select select_food_tteeeeeeeeeeeeee too..tte.eeeettp.......
return return
def attraction (trpeicn target_city):
attractions Attractions
sclect_attr_tyye
attracticns.select
target_city, key="name",
attraction -type
the type of at- traction in the
func=1ambda x: == ["type"]
target city
not select attr eestr_type.empty.
return select_attar itppe.eiiottttt.lollcc.oooo
return
def accomnodation accomodlaiicoottppeeatttttii target_city):
accomnodations Accommodations ()
select_hotel_type
accommodations.cd selectt
accommo- dation_type
the fcature of accommo- dation the target city
target_city, key="name",
func-lambda x: -- activity activity("positions
" ""featurehcteltypee)
not boooc__hhtte_ttpeemmmtt
return select_hotel iloc
return
Table 12: Concept Function
25

---

Published as a conference paper at ICLR 2026
Query in Chinese (from subset): NM - A fF of W #2 1
Current location: Chengdu. My friend and want to go to Nanjing days. We need twin room in hotel where play cards. Please provide travel plan for us.
accommodation type set-set()
for activity allactiivities(lamm
icativvity type(activity) 'accommodation'. accommodation type aldoemmaaddacoommmaaiiionn typefactivity, target city(plan))
result=[ (1 [F" <raccommodation type
Query in Chinese (from medium subset): A. H
19 BL 3000 A
Current location: Chengdu. want to travel alone Chongqing for days a budget of 3000 RMB. plan to take
the train, want to cat hotpot, and visit Hongya Cave.
total cost-0
activity in allactiviities(pann)
total cost activiyy cost(activity))
cost innercity transport cost(activity transports(activity))
result=(total costC=3000)
Intercity transport set set(
activity allactivvitiesplan))
l'activity type(activity) in I'train'. 'uirplane']: intercity transport set.addiintercitt transport typefactivity)
result=( ("train)==="" inteeriity transport set)
restaurant type sep__setsse((
for activity in allacttivttiesplam):
if activivtty sypo(activity) ['breakfast', 'lunch', 'dinner)]: restaurant type set.addfrestaurant type(activity, target_ city(plan))
""uu==(("""""<<<rttuuuu -=rrestarrant type set)
attraction name set()ooo activity in allactivities(plan)
activity type(activity) 'attruction': attraction name actiadi(attivity position(acttiitty)
result=(/ ('85 " cc=ttractioon name set)
Query Chinese (from human subset): [ HT
2.
R1 AS AURR
Aitix
36 3 R. E
E 14 Ptx 1% fj db S
2
English translation: (Current location: Nanjing, Destination: Wuhan, Number travelers: Travel days: 3] The two of us want to visit Wuhan for days. We mainly want to experience some of the historical areas Wuban and also try the local specialty foods that residents often cat. How should we plan our itincrary? attraction type sct=sct()
for activity in allactivities(plan): if f'activity
aartaattiiiiiy=aaatttttttiio attraction type attratiotn typolactivity, target city(plan)))
result=( (1Dj @ 21
type sct)"
restaurant type set sel() I'ffor activity in allactivities(plan):
if aattiiity typo(activity) in l'brcakfast", 'Tunch', 'dinner];; restaurant type addereseetttrr typelactivity.
aagget city(plan))
jesult=( ('50 #e) type sct)"
Query in Chinese (from human subset): [ [ S R -1 (i& 3231.
|6j00 003 2000.
[Current location: Nanjing, Destination: Ilangzhou, Number of travelers: 2, Number of travel days: 3] We plan to visit West Lake Hangzhou with budget 2000. Please provide me with travel itinerary.
attraction name set=set()
activily in allactivities(llan)
if activity - attraction': attraction name sct.addractivity positionlactivity)) result=( (116)))) Bi #t X atatraction name set)
c
for activity in allactivities/ plan))
lotal cost activity cost(activily)
total cost innereity transport cost(activity transport((activit))
(sota= cost<=2000)"
Figure 10: Examples of travel requirements and their DSL expressions.
26

---

Published as a conference paper at ICLR 2026
Function Name
Meaning
Implementation
def innercity__rranpprr__vpe (transports);
len (transperts)
innercity-- transport.-tyy type
type of innercity trans- port
the
return transports [1] ["mode"]
elif len (transports) == 1:
return transports [0] ["mode"]
return
intercity.-
the type inter-
def srrpetty_transppttcyeeelllaae
return activity.get "")
transport-ttype
city transport
def tneerdtisy.rangppos..tttttttttt (ttrrsrrrttiieeeerrtrrsssrttst (transports) return transports (Mstttttttttststttttttiiii
innercity
the start time innercity trans- port
transport-
start_time
def cnttercit__trrnsporr_enddd (trarn return transports sand_time""
innercity
the end time of innercity trans-
transport- end. time
port
def prrecttyytranspott_orioco (erteree (activity); "start" activity:
intercity Lraggn
the origin city of intercity transport
for city in city_list:
if city in aetivvity(sstart)) return city
return
def intercity
intercity_transpprtt_ooo transport_destiineeeee
(activity)
"end" in activity:
intercity.- transport.- destination
the destination city of intercity transport
for city in city_list:
if city in activity ("end")
return city
return ""
Table 13: Concept Function
D.6 DSL. EXPRESSION FOR PREFERENCE
We introduce six common preferences from surveys to construct the preference sub-datasets.
Table provides a summary of these preferences.
The corresponding DSL could be formulated follows.
The number of attractions visited
count for act_i in all (ptivitess activity_type (act_ =aattraction; count count + return count
27

---

Published as a conference paper at ICLR 2026
Example of Prompts for Data Generation
You are a user who wants to ask an AI agent to help you natural language
plan trip. Please construct inquiries based on the following example and provide the corresponding logical constraint expressions, Note that tickets" and "people _umberr are the same.
Example:
JSON:
(}
Use the following restaurants.
Restaurant name: {}
This means that "restaurant_names" should include this restaurant.
The dining options may not always be exactly described by the provided features; synonyms can be used. For example, if the hotel's feature is pool, you could ask naturally in language like "I want to swim in the hotel pool." "
Now, your departure location is {,, and your destination is
(). The number of people is Or and the number days is ().
Now please provide a JSON inquiry. JSON:
Figure 11: An example of prompts for data generation. This example is about restaurant name. By replacing this with other constraints combining multiple constraints, generate data with different levels of difficulty based on different constraints.
The average travel time between POIs cost transport_cone for activity in allactivities (plan)
transports
activvit__transpots (activity) transport_count +-
time_cost +- imeeriity_trrnssprrtiiimm
transports)
ewerage_time_coe time_cost transport_count transport_couue else -1 return avcrage_time_cost
The average travel time to restaurants restaurant_coont =
time_cost =
for activity in (platti
if prrittyyype(aattivie in ('breakfast', (lunch', 'dinner') )
restaurant_coume +=
time_cost += activity)) if restaurant_count 0:
average_time_com -1
else:
average_time_cost time_cost restaurant_count
return average_time_coe
The ratic of food cost
foot_cost
28

---

Published as a conference paper at ICLR 2026
total_
for activity in aplastivitiessplan):
total_cost += activity_cost (activity) total_cost
if activity_.tye
in l'breakfast', 'lanch', 'dinner');
foed_cost activity_cost (activity)
Food_cost_ratie fosd_cost tetal_cost total_cost > else -1
return good_costtratble
The cost of accommodations Accommodation_coss =
for activity in (plactivitiessllan)) (plan) if activity_type (astivity) aaccommodation':
accommodatiin_coost +- activity (activity)) return accommodation_com
The average distance to poi (e.g. xxx)
target_pod 'xxx'
poi_list list()
total_distanee
poi_count=0
city = target_cityy(pla))
for activity in allactivitie (plan) (plan)
if activity_type (astivity) in ['breakfast', 1unch", 'dinner', accomrodation! (attraction") poi poi_list www.lisstpppan..... for poi in pei list:
position (activity))
total total_distance distance poi distancecnccityy target_poi poi) poi_count average_dist_cost total_distance / poi_count if pol_count 0 else -1 return @werage_dist_com
D.7 BENCHMARK DIFFICULTY AND APPLICABILITY
While the Human subset presents significant challenges, the baseline NeSy solution has achieved 60.6% and 46.7% FPR on Easy and Medium subsets, respectively, providing developers with action- able validation points for initial testing and refinement. Additionally, the Human subset's extreme difficulty arises from open language reasoning and unseen concept composition, key challenges absent in prior benchmarks but unavoidable in practice. By explicitly formalizing these challenges, ChinaTravel has provided a roadmap for advancing agents toward real-world robustness. Despite current LLMs' limitations in handling unseen combinations, their success in code generation suggests that post-training with DSL may enhance their understanding of diverse travel needs, moving toward real-world applications.
D.8 DSL EXTENSION
The design DSL a modular, domain-agnostic framework whose core operators are reusable beyond the current instantiation. Concretely, it separates generic compositional operators, logical, arithmetic, set, and temporal constructs, from pluggable library of domain-specific predicates and attribute-access functions. Extending the constraint library to include new concepts, such as 'a scenic rating of a route' or 'avoid areas with high COVID-19 cases", is straightforward, two-step. incremental process, not framework overhaul.
(I). Sandbox extensio. Integrate the new attribute into the sandbox by adding a corresponding field to the relevant entities. For example, to support scenic beauty of route, could add a numeric scenic eeratttt ingg attribute to attraction entries, to model avoid areas with high COVID- cases, can add boolean covid.a risk field to POIs.
(2) DSL function definition. Expose this attribute through a small helper function or predicate the DSL library (e.g., get scenic rating (attract. on) get covid risk (POT) User
29

---

Published as a conference paper at ICLR 2026
Examples of Generated Data
Example
"start_cit"": "bt/
"target.city":. "17""
"hard "hard_logic:: "days=2",
"people number==1",
"tickets==",
"I "(" v food_type"
1.
"nature language": T
Example
"start.city": ""
"target city": "L""""
"hard logic":
"days=2",
poopllcnumber=="",
"intercity_treee ['ralpprtt==(ainlanne]]"" "tickets=33",
"rooms==3". "room type==1=
1.
"nature " UA GAL
fr. ハ i 1
Example 3
"start.city":: "
"target city": "T
"hard logic":
"days=3"
"cost<7300".
"{" "(" < food_type",
"intercity-tranae tranritytttanpport=tttmmnn"
"tickets==3",
"rooms==2", "room_tpe==2"
J.
"nature :antuagel A BHENE
t.. A 7300T,,
Figure 12: Examples of Generated Data
30

---

Published as a conference paper at ICLR 2026
requests such "prefer scenic routes" and "avoid covid risk" can then be rendered as constraints such as:
maximize serniicscore_surr as a seft preference
vcenic_score_sunes
for act_i i in all Lattivitie if activity_type oc activity_position (act_ii) return scenic score sum
scenic_score_ste +-
avoic covid risk risk_flag flag
hard constraint
for act i in all activities
risk_flag +- ris_ (activity_positioon (act_i)) return trisk_flag==O)
These two steps correspond exactly to make the information available and textbfprovide a way for the agent to query it. They are both textbfnecessary and close to minimal, changes are required to the core DSL grammar, compositional operators, planner, or verification engine.
Moreover, new concepts naturally be combined with existing temporal and structural concepts to express richer user requirements, like "visit the most scenic attraction in the itinerary on day avoid COVID-risk restaurants day 1 and COVID-risk attractions day 2'. ChinaTravel is explicitly designed to make such user- eser-friendly, open-ended compositional constraints representable and automatically checkable, and we hope this will draw the community's s attention to these more realistic forms of constraint-aware LLM agents.
E DISCUSSION WITH TRAVELPLANNER
TravelPlanner's logical constraints contain the total cost, 15 cuisines, 5 house rules, room types. and 3 intercity transports. ChinaTravel's logical constraints contain the total cost, 42 cuisines, 15 attraction types, 78 hotel features, 2 room types, 2 intercity- transports types, 3 inner-city- -ttarrsstttr types, and specific POI names (attractions, restaurants, hotels). Crucially, our benchmark introduces compositional constraints derived from human queries (e.g.. "return before 7 PM", "cost of intercity transports"), reflecting real-world complexity. The key advancement lies in addressing open- open-language reasoning and unseen concept composition, which represent significant challenges beyond Trav- elPlanner's 's scope. Our Domain-Specine Language (DSL.) enables automated validation of these combinatorial requirements, bridging the gap between synthetic and real-world needs.
We also provide some example queries and corresponding examples from the TravelPlanner at each level in Figure 14 and
As shown in Figure 13 in the "easy level", TravelPlanner only includes constraints on cost. In contrast, China'rravel demonstrates significant advantages over TravelPlanner, particularly in terms of personalized support for specific Points of Interest (POIs) and more realistic transportation and time management. These advantages are crucial for developing and evaluating language agents that can handle real-world travel planning scenarios effectively. China allows users to directly specify POI names, such as "Nanjing DaPaXiang" or "HuQiu Mountain Scenic Area," requiring the agent to precisely match the entity information from the travel sandbox.
As shown in Figure 14] in the medium set, TravelPlanner includes queries with types of constraints: cost and cuisine, or cost and accommodation. In contrast, ChinaTTravel includes queries with 2-5 constraints, reflecting more complex and diverse multi-constraint requirements. This difference highlights the ability of China to handle more realistic and varied travel planning scenarios.
As shown in Figure 15 TravelPlanner includes queries with multiple constraints, such as cost, ac- commodation type, and cuisine preferences. However, ChinaTravel goes step further by including queries with unseen logical constraints and more colloquial expressions. These unseen logical con- straints and colloquial expressions essential for travel planning agents handle real-world users effectively. They reflect the complexity and diversity real-world travel planning scenarios, where users may have diverse requirements that need to be understood and addressed. By incorporating these
31

---

Published as a conference paper at ICLR 2026
elements, ChinaTravel bridges the gap between current academic research benchmarks and real- world application demands, making it a more comprchensive and realistic benchmark for evaluating the capabilities of travel planning agents.
Here, we further provide a performance comparison across TripPlanning Zheng et al. 22024 Trav- elPlanner Xie et al. 2024 (Val-180) and our ChinaTravel (Human-155) 154).
Method
Model
TripPlanning
TravelPlanner
ChivaTravel
Pure-LLM
37.1
4.44
2.59 0 (GPT-44)
GPT
31.1 (GPT-4)
4.4 (GPT-4-Turbo)
NeSy
TTG(DS-V3)
91.7
1.29
LLM- LMM-ModuloDDS-VV)
98.5
25.55
2.59
Table 14: Performance comparison across benchmarks.
As shown in the Tab. 14 we could find that: (1) Catastrophic Failure of Pure LLMs: While Pure LLMs show decent performance on TripPlanning (DeepSeek 37.1%, GPT-4 31.1%), a pure reasoning task, their success rate dramatically drops to around 4.4% when tool-calling is introduced in TravelPlanner. Moreover, when facing the compositional complexity and open-ended nature of ChinaTravel, LLM performance collapses to near-zero (c.g., GPT-4o achieves 0%) ). This highlights that both TravelPlanner and ChinaTravel poses an agentic challenges, which existing L.LMs cannot handle, as we claimed in the paper. (2) Failure of SOTA Neuro-Symbole Methods: TTG excels on TravelPlanner (91.7%) because its symbolic logic is good fit for TravelPlanner's fixed and predefined constraints. However, TTG's s success rate plummets to 1.29% on ChinaTravel. As we analyze in Sec. 4.2 and Fig. 6a, this confirms that TTG's constrained symbolic system cannot generalize to long-horizon planning required by ChinaTravel. LLM-Modulo demonstrates improvement over pure LLM on TravelPlanner (4.4%~25.55%)) China'Travel (2.59%). This again validates our argument: ChinaTTravel is merely harder version of existing benchmarks, it requires a new level of planning difficulty that current SOTA methods lack. These results unequivocally confirm that ChinaTravel introduces a new open-ended dimension that exposes the limits of both pure LLM and current neuro-symbolic agent designs, thus strongly validating its contribution as novel, challenging benchmark.
via symbolic constraint feedback, but still fails on
F NESY PLANNING
Since the Z3 solver from (Hao et al. 2025) would restructure the tool API to return travel information expressed in specific Z3 variables, which may not be feasible given that APIs in the real world are typically black boxes that agents can only call. Following their two-stage solution, we first extract logical constraints from natural language. Based on these constraints, we implement a step-by-step plan generation process using depth- ferst-first search, mimicking how humans plan to travel by arranging activities one by one. As shown in Fig. 5 we first translate the natural languages to logical constraints through prompting. generate the next activity type based on the current plan, and then recursively generate the next activity until the goal is reached. The generated plan is then used to solve the problem. In the second step, we define the rule-based activity selection and score function. For example, if the current time is in the [10:30, 12:30] and there is no scheduled lunch in the current plan, then the agent should find a restaurant have lunch at this time. If the current time is after 22:00 and there are no open-time attractions nearby, the agent should choose to return to the hotel. For the score function, we select the restaurants that satisfy the required cuisine and sort the candidates by the price if there budget constraints in the constraints C. These ranking functions will help to find a feasible solution as soon as possible. In ChinaTravel, the duration arrangement of activities continuous and difficult to enumerate and search. We pre-define meal or visit to attraction as 90 minutes, and when there are less than 90 minutes until closing time, the event continues until the closing time. Given these designs, adapt the neural-symbolic solution into multi-POI planning problem and evaluate it in the ChinaTravel benchmark.
Given that some queries particularly challenging due to the limited number of feasible plans, we set the maximum runtime for the symbolic sketch from interactive search to minutes per query, excluding the L.M inference time, to ensure fair comparison across different models. If
32

---

Published as a conference paper at ICLR 2026
Algorithm 1 Depth-First Greedy Search
Require: Constraints C, current plan p,
if the least activity is an intercity-transpen from destination to origin then
return
C), p
> The plan p is finished, return the validation result.
end
type GetNextActivity Type(p)
Select the next type of activities, e.g. lunch, attraction. b Collect the corresponding information for the activity type
candidates = (oypes)(type)
scores LLMMScrrecanddddatsss p. C)
> Score candidates through constraints C.
for activity in candidates do
p.push(activityy flag, p= if flag then
b Perform greedy search with priority ranking.
p)
return True, P end if p.pop(activity) end for return False,
> Return the solution p if the validation is passed.
Fail to find a solution with the given conditions.
a plan satisfying the generated DSL validation found within the time limit, it is returned directly. Otherwise, the program halts when the time limit is reached, and the plan that satisfies environmental constraints while achieving the highest number of validation code successes among all intermediate results is returned. In cases where no environment-complaan plan is identified, the partially completed plan generated up to that point returned.
In the Figure
19 20 and 21 we provide the prompts of the LLM POI-ranking phases.
G EVALUATION METRIC IN COMPETITION
The Delivery Rate (DR), Environmental Pass Rate (EPR), Logical Pass Rate (LPR), and Final Pass Rate (FPR) have been detailed in TravelPlanner (Xie et al. 2024). To make the paper more self-contained, we also provide the corresponding definition here.
This metric assesses whether agents can successfully deliver a formatted plan. For the Nesy planning. if a solution that satisfics the logical constraints has not been found by the time out, the search is terminated, and the current solution that satisfies the environmental constraints returned. If no solution that satisfies the environmental constraints is obtained, an empty plan is returned. Therefore, unlike the pure LLM method, which primarily assesses the Delivery Rate based on whether the output meets the formatting requirements, the nesy planning method, which uses depth-first-search to arrange POIs one by one, shows differences in the Delivery Rate. These differences mainly reflect the proportion of effective solutions obtained within a limited time based on the LLM's POI recommendation. This proportion demonstrates the degree which the LLM prioritizes POI arrangements from natural language perspective and meets formalized logical requirements. The more accurately the L.M can arrange POIs that are beneficial for long-horizon planning, the more likely it is to obtain effective solutions and improve the Delivery Rate.
Delivery Rate:
Environmental Pass Rate Comprising the environmental dimensions (as detailed in Tab. (7). this metric evaluates whether language agent can accurately incorporate sandbox information into their generated plans.
EPR micro Epep Eee Eng (.ps
P
Enwl
EPR - macro = Epep Iee Env passedde..))
PP
33

---

Published as a conference paper at ICLR 2026
Logical Pass Rate Comprising the logical dimensions (as detailed in Tab. 8). this metric evaluates whether a language agent can accurately meet the personalized requirements of the queries.
Epep Exec, passedd(p..p)
LPR - micro LPR-m
SpepICp,
LPR macro Epep IoEE, Apassed(Cy..))
|P]
Final Pass Rate This metric represents the proportion of feasible plans that meet all aforementioned constraints among all tested plans. It serves as an indicator agents' proficiency in producing plans that meet a practical standard.
FPR Eper passedd Ene.p) passed((,..))
PP
Table 15: Multi-Preferent Comparison of BQ and PEQ.
Preference Combination
Vaulc-1
Vaule-2
Rank-1
Rank-2
Agg. Rank.
BQ
PEQ
BQ
PEQ
BQ
PEQ
BQ
PEQ
BQ
PEQ
PO 1, P1 &
0.79
0.83
28.0
29.7
1.44
1.55
1.44
1.55
1.44
1.55
PO 1, P2
0.82
1.26
29.0
31.9
1.56
1.43
1.43
1.56
1.5
1.5
PO 1, P3
0.81
0.94
0.18
0.20
1.42
1.57
1.59
1.40
1.51
1.48
PO 1, P4
0.79
0.97
1221
441
1.46
1.53
1.73
1.26
1.59
1.40
PO f. P5
0.78
0.91
33.6
34.0
1.37
1.62
1.70
1.29
1.54
1.45
4, P2
28.2
27.8
26.6
30.1
1.62
1.37
1.51
1.55
1.44
P3
28.2
36.2
0.20
0.27
1.31
1.68
1.4
1.45
1.54
P1 J. P4..
30.3
44.8
1440
585
1.14
1.85
1.77
1.22
1.45
1.54
P1 P5
30.1
38.3
30.7
30.2
1.27
1.72
1.69
1.30
1.48
1.51
P2 J, P3
24.7
23.3
0.27
0.27
1.43
1.56
1.60
1.39
1.52
1.47
P2 J, P4.
24.1
21.1
1687
719
1.51
1.48
1.89
1.10
1.70
1.29
P2 J. P5
28.0
30.8
29.4
26.0
1.51
1.48
1.89
1.10
1.70
1.29
P3 1, P4
1..88
0.26
1229
531
1.64
1.35
1.69
1.30
1.66
1.33
P3 1, P5
0.22
0.22
33.3
29.0
1.51
1.48
1.84
1.15
1.68
1.31
J, P5
1366
767
33.1
31.6
1.67
1.32
1.45
1.54
1.56
1.43
Aggregated Ranking
1.56
1.43
H ADDITIONAL EXPERIMENTAL RESULTS
H.1 RESULTS WITH LARGE REASONING MODEL
The current experimental results have covered Qwen3-8B, the largest CoT-eenaleed reasoning models we could feasibly run within our local computational resources. We have further conducted the additional experiments with DeepSeck-RI and DinSSsck-R--Disiil----eeee
Note that R1-Act is inherently a reason-then-eeeee paradigm. The results show that pure-neural methods still struggle on ChinaTravel. Interestingly, DeepSeek-Ri RI does not consistently outperform RI-Distill- Qwen-7B. From the observation over experiments, one plausible reason that R1 tends to over-think, which weakens final instruction-oolllwiiii in long contexts and yields unsatisfactory performance. Encouragingly, even 7-8B LLMs already exhibit some ttanslaatin ability, so the community can conduct cost-effective post-training research on ChinaTravel with modest resources.
34

---

Published as a conference paper at ICLR 2026
Table 16: Results on the Easy and Human- 154 subsets.
Easy
Human-154
Method
Model
DR
EPR
LPR
C-LPR FPR
DR
EPR
LPR
C-LPR FPR
Mic. Mac. Mic. Mac.
Mic.
Mac. Mic. Mac.
Act
RI
43.3 31.6 2.9 39.2 24.9
2.7 2.9
38.0
21.1 0.0 33.3 15.5
0.0 0.0
NeSy
RI-D. H...D.Qwe-77e
7B.53.3 53.3
53.3 49.7 38.7
49.7
49.5
59.1
58.8
52.0 51.3
45.2 28.6
NeSy
RI
58.3 58.3
58.3 53.7
32.6
53.7 32.6
46.8
46.6
45.5
39.2
38.0 23.4
NeSy oncle RI-D.-Qwent -ew.D-----nnBB 63.7 63.7 63.7 61.3
53.7
61.3 53.7
40.9
40.8
39.0
36.0
34.3 30.5
NeSy oraele RI
50.0 50.0 50.0 49.8 46.3
49.8 46.3
43.5
43.5 42.9 39.7 33.8
39.2 33.8
H.2 RESULTS ON MEDIUM SET
For organizational coherence in the manuscript, we elected not to include medium- medium-complextt complexity experimental results in the main text. The medium set features user inputs containing 3-5 logical requirements, representing the mid-range complexity tier that bridges simple queries and the highly complex open-ended scenarios.
Table 17: Results of different LLMs and planning strategies on the ChinaTravel medium subset.
DR
EPR
LPR
C-LPR FPR
DR
EPR
LPR
C-LPR FPR
Mic. Mac.
Mic. Mac.
Mic. Mac.
Mic.
Mac.
72.7 52.3
0
63.5 15.3
0
71.3 71.9 69.3
69.4
50.0
69.3
46.7
Act
97.4 70.5
89.3
55.3
NSP
68.0 68.0 68.0
64.1
46.6
64.1
46.7
ReAct
35.2
37.6
4.0
0
S>
53.3 45.9 16.0
49.2
33.3
14.8
8.50
(zero shot)
92.0
0
78.6 22.7
0
68.6 65.4 54.0
66.2
61.3
52.5
54.0
NSP
ReAct
87.7
3.33
82.6
48.7
2.95
1.33
60.8 59.4 54.9
60.3
58.2
60.3
56.9
oracle
(one-shot)
94.7 69.2
0.67
91.8 64.0
0.53
0
53.3 51.3 36.6
51.9
43.3
34.8
34.6
H.3 MULTI-PREFERENCE COMPARISON
For multi-preference scenarios (e.g., balancing "attraction visits 1" and "transport time 4"), we adopt an averaged aggregation approach, where rankings reflect the combined performance across all preferences. This framework ensures scalability and objectivity.
To rigorously evaluate the ability of language agents to balance multiple soft constraints, we con- structed 15 test subsets by pairing six user preferences (PO-P5) into all possible combinations (e.g., "PO P1"). Each subset contains queries with two preference requirements. We compared two meth- Query (PEQ). by quantifying their performance through DSL--based Preference Ranking metric. For each subset, extracted numerical scores for both preferences ( and Value-2), computed individual rankings (Rank-1, Rank-2), and derived an aggregated ranking (Agg. Rank.) to reflect overall performance. The results are provided in the Table [5]
ods, Baseline Query (BQ) and Seeferenne-Enhhanee
From these results, could find that: (1) PEQ Outperforms BQ in Most Scenarios: In 10/15 combinations, PEQ achieves superior aggregated rankings (Aggregated Ranking = 1.43 vs. BQ's 1.56). Notably, PEQ demonstrates stable improvements on preferences P3 (e.g., maximizing dining qualityf) and P4 (c.g., minimizing accommodation costs.). For instance: In "POt + P4."", PEQ reduces accommoaation costs by 64% (Value-2: 441 vs. BQ's 1221) while maintaining high attraction counts (Value-1: 1: 0.97 vs. 0.79). For "P3t + P4J", PEQ simultaneously improves dining quality (Value-1: 0.26 vs. BQ's 0.18) and lowers costs (Value-2: 531 vs. 1229). This stability likely stems from the direct impact of POI selection these preferences. LLMs in PEQ effectively prioritize low-cost hotels or high-quality restaurants through natural language hints (e.g.. "reduce the cost on accommodationn"" enabling explicit alignment with P3 and P4 requirements. (2) Challenges in Balancing Multiple Preferences: The results also reveal inherent difficulties harmonizing conflicting preferences, particularly when optimizing requirement necessitates sacrificing another.
35

---

Published as a conference paper at ICLR 2026
Notably, in the POt PI scenario, PEQ underperforms BQ on both preferences, highlighting the inherent difficulty in resolving conflicting objectives. While PEQ marginally improves attraction counts (Value-1: 1: 0.83 vs. BQ's 0.79), it incurs 5.7% increase in transport time (Value-2: 29.7 vs. BQ's 28.0). This trade-off results in a worse aggregated ranking for PEQ (1.55 vs. BQ's 1.44), indicating that the combined effect of conflicting preferences negates the benefits of natural language guidance. In 9/15 combinations, PEQ improves one preference at the expense of the other. For example: P14 + P4J: PEQ reduces accommodtiion costs by 59% (Value-2: 585 vs. BQ's 1440) but increases transport time by 48% (Value-1: 1: 44.8 vs. 30.3). The inability to concurrentyy satisfy both preferences underscores the limitations of current LLM--riven prioritization in handling trade-offs.
Our experiments demonstrate that the neuro- symuo-yymbolic agent (PEQ), enhanced by LLM-driven POI recommendation, outperforms baseline methods multi-preference travel planning. By integrating natural language hints to guide POI selection, PEQ effectively translates user requirements into actionable itineraries, demonstrating its capability to handle synergistic preferences. However, balancing inherently conflicting objectives remains challenging. This highlights the need for future advancements, such as domain-specific fine-tuned LLMs to better resolve preference conflicts or multi-objective optimization techniques to systematically navigate trade-offs.
H.4 ANALYSIS OF PURE-LLM METHOSS
Pure L.LM-based methods have demonstrated significant shortcomings in constraint satisfaction, as evidenced by their near-zero success rates in benchmarks like TravelPlanner. We also attempt the multi-round refinement methods like Reflexion. While theoretically promising, it still impractical in our context. In preliminary evaluations, Reflexion not only failed to achieve improvements in constraint satisfaction (consistent 0% FPR) but also incurred prohibitive computational costs due to its reliance iterative token-heave interactions. This rendered large-scale evaluation infeasible given our resource constraints. In light of their current limitations in constraint satisfaction, NeSy frameworks remain the effective pathway for real-world travel planning. Therefore, in the main body of this work, we mainly analyze the Nesy method.
In this section, further summarize the key failure modes of pure- Lur methods observed in our experiments:
(1) Incorrect API Calls: LLMs frequently generate invalid hallucinated API calls, leading to cascading errors in downstream planning. For instance, models may query non-existent APIs (e-g., city_ city stransporrtsseeett instead of seterriitt_ransoorttste attractions by an unsupported feature like "bus"). Such errors exhaust API call limits and prevent agents from retrieving essential information.
or misuse parameters (e.g., filtering
(2) Repetitive Output Loops In iterative planning frameworks like ReAct, LLMs often enter infinite loops when resolving constraints. For example, an agent might repeatedly query transportation details for all candidate attractions, even after selecting one, due to failure to update its internal state. This behavior mimies the "hallucination loops" reported TravelPlanner paper.
(3) Reasoning-Action Inconsistency. In ReAct framework, the model first reasons and then takes an action. However, the reasoning and the action are not always consistent. For example, the model may reason that the user wants to book flight, but then take action to check the information of trains. Another example is that the model may detect that the expenses exceed the budget but does not respond to this and ultimately generates plan that exceeds the budget.
(4) Critical Information Missing, Even when intermediate steps (e.g.. API responses) are logged in "notebook." LLMs frequently omit essential details when synthesizing final plans. A recurring failure is neglecting return transportation (c.g., omitting the train from Shanghai back to Beijing), which violates feasibility constraints.
Figure 16 provides the fail examples of ReAct (one-shot) with DeepSeek, which outperforms other pure- LLM based methods in the main experiments.
These limitations underscore the inadequacy of pure-LLMMbaasee approaches for deployment in long-horizon and constraint-rich domains like travel planning.
36

---

Published as a conference paper at ICLR 2026
H.5 THE ABLATION STUDY OF THE PROPOSED NESY PLANNING
The Impact of Iterative NL2DSL Translation. Tab. B explicitly compares NeSy Planning with or without oracle translation. This quantifies the translation module's impact, i.c. is critical but currently bottleneck due to unseen concept composition (as shown in the Fig. 7b).
The Impact of Symbolic Search Sketch. In NeSy Planning, the symbolic search sketch uses DSL constraints guide sequential construction with backtracking, whereas the L.M-modulo baseline only applies the same constraints for post-hoc error correction without search. LLM--ooulo serves ablation. As result, this search-based decomposition turns constraint feedback into much more effective plan refinement.
Impact LLM-driven POI Ranking. We ran NeSy and NeSy(Oracle) with random POI ranking while keeping all other components unchanged. As shown in the Tab. 18] this leads to large and consistent drops in FPR: on the easy split from 74.0 30.3 and 52.6 > 25.6, and on the human split from 45.4 - 38.3 and 37.0 + 31.8. These results indicate that the LLM--drienn ranking makes a substantial contribution by steering symbolic search toward semantically appropriate POIs. At the same time, even the random-ranking NeSy variants still significantly outperform pure-LLM agents in Sec. 4.2, suggesting that POI ranking an important but not sole factor and that NL2DSL translation plus symbolic search are also crucial to the gains of NeSy Planning.
Easy-300
POI-Ranking
FPR
Human-154
POI-Ranking
FPR
NeSy
LLM
52.6
NeSy NeSy
LLM
37.0
NeSy
Random
25.6
Random
31.8
NeSy(Oracle)
LLM
74.0
NeSy(Oracle) NeSy(Oracle)
LLM
45.4
NeSy(Oracle)
Random
30.3
Random
38.3
Table 18: Comparison of FPR across Easy-300 and Human-155 under different POI-Ranking methods.
H.6 RESULTS ON ENGLISH SETTING
We have extended Cravel English and is now a multilingual benchmark resource, making it convenient to global researchers and facilitating comparability. In the Tab, [9] we provide the preliminary validation on easy-300 and human-15: 154. The results confirm that the fundamental challenge raised by ChivaTravel is
Method
LLMs DR
EPR
LPR
C-LPR FPR DR
EPR
LPR
C-LPR FPR
Mic. Mac. Mic, Mac.
Mic. Mac. Mic. Mac.
ReAct
92.3 53.6 2.33 77.2 39.3
2.15
2.0 78.6 52.5 0.65 78.1 42.9
0.88
0
NeSy Planning
81.9 81.3 77.2 57.7
76.6
57.7 59.7 59.1 57.8 51.5 41.6
49.6
40,9
75.7 75.1 75.0 70.0 49.7
69.6
49.7
49.3 49.3 47.4 41.4 33.8
40.2
33.8
77.0 74.4 40.3 70.2 48.3
37.6
41.6
39.6 36.7 26.6
34.9
26.0
NeSy Planning
76.7 76.7 76.7 73.5 63.7
73.5
66.3 68.2 68.1 66.2 59.8 51.9
57.7
51.9
Oracle Translation
79.7 79.7 79.7 76.7 67.3
76.7
67.3
53.9 53.8 52.5 44.6 40.9
43.8
40.3
77.0 77.0 77.0 73.9 62.3
73.9
62.3 61.0 61.0 59.7 52.2 43.5
51.1
43.5
Table 19: Results Easy-300 and Human- 154 from ChinaTravel-EN.
From the results, we could find that, the performance of pure LLM methods on long-horizon agentic plamning remains 0% in the English setting. This validates core finding that LLMs fundamentally struggle, regardless the sandbox language. Moreover, the results of NeSy methods, could find the DSL translation bottleneck is still essential for grounding complex constraints.
37

---

Published as a conference paper at ICLR 2026
H.7 ANALYSIS OF THE CHALLENGE ON COMPOSITION COMPLEXITY
We further conducted experiments Human-1000 investigate how model performance scales with the complexity of composition. Following the compositional generalization community, we define composition complexity (C) the number of basic concepts involved in a DSL requirement. Specifically, we evaluate the matching rate (%) of POI Reasoning (correctly mapping user intent to specific POI requirements) and Syntax Generation (correctly translating query to the POI-masked DSL syntax) as the number of constraints (C) increases from to 5. The results are provided in the Tab. 20
POI Reasoning
Syntax Generation
method
C=1
C=2
C=3
C=4
C=5
C=1
C=2
C=3
C=4
C=S
DeepSeck-V. V3
100
100
83.9
80.7
50.0
63.9
0
2.2
0
GPT-4o
100
100
63.9
59.0
24.9
46.5
0
9.1
0
0
Qwen3-8B
39.2
0
0
Table 20: Challenge Analysis with different constraint numbers C.
The experimental results clearly show that agent performance degrades significantly as the number of composed concepts (C) increases. This finding is consistent with general observations in the compositional generalization community, These results also further provide C-dependent evidence for our core claim: the compositional challenges introduced by ChinaTrraeel, both in syntax structure and semantic understanding. represent fundamental bottleneck for existing LLMs.
Performance Degradation with Composition Depth:
STATEMENTS ABOUT SCIENTIFIC ARTIFACTS
The ChinaTravel benchmark is designed to facilitate research in natural language processing and artificial intelligence, specifically for travel planning tasks. China Travel includes a travel sandbox, user queries, and an evaluation framework intended for non-commercial, academic research purposes.
Availability.
We will publicly release the ChinaTravel benchmark upon publication to facilitate community research. We look forward to broader adoption and extension of this benchmark.
Licenses. The ChinaTrave benchmark and its associated datasets are licensed under the Creative Commons AttributionnNNnnComneee Commercial 4.0 International (CC-BY-NC 4.0) license. This license allows for the free use, distribution, and reproduction of the benchmark in any medium, provided that appropriate credit given the original authors and the source of the data is acknowledged, and that the use is for non-commercial purposes only.
Data anonymization and offensive content. We anonymized the human queries during collection and instructed participants avoid including sensitive information. We removed queries containing offensive content during the data cleaning process.
J STATEMENTS ABOUT HUMAN PARTICIPANTS
For the collection of Human- 154,, we recruited over 250 volunteers through structured questionnaire to collect authentic Chinese travel requirements. Participants were informed about the public use of their data and instructed to avoid including sensitive personal information. During data cleaning, offensive content and identifiable details were removed. While no explicit ethics board approval is mentioned, ensured compliance with anonymization practices and obtained participant consent for data inclusion. The final dataset contains 154 human derived queries reflecting diverse real- wolldwrldd travel needs.
38

---

Published as a conference paper at ICLR 2026
J.1 INSTRUCTIONS GIVEN To PARTICIPANTS
To gather the authentic travel requirements, we collected data through a carefully designed question- naire. We provided the following instruction information to the participants:
The specific constraints the agent can handle and the corresponding details, including the types and specific names of attractions, restaurants, and hotels; requirements for intercity transportation (airplane or train) and urban transportation (walk, taxi or subway); as well as budget limitations for overall expenses or specific activities (such accommodation and intercity transportation).
2. The necessary information should be provided in the query, including the departure and destination cities of the trip, the number of travel days and constraint information.
3. A detailed example with the query and travel planning response.
Fig. 17 and Fig.
18 respectively show the questionnaire and its translated version.
J.2 RECRUITMENT AND PAYMENT
For the collection of 1554, we recruited a total of 250 student volunteers to provide authentic Chinese travel requirements. The participants included 121 undergraduate students, 86 master's students, and 43 doctoral students. The task of understanding the query background and providing travel requirements was estimated take 1-2 minutes per participant. Given the simplicity of the task and the fact that it did not require extensive professional background or expertise, we compensated each participant with CNY. This compensation was deemed adequate considering the nature of the task and the time required to complete it. The payment was determined based on the estimated time and the straightforward nature of the natural language requirements, ensuring a fair and reasonable reward for the participants.
For Human- 1000, we partnered with WJX (a professional survey platform) to scale data collection. Each valid query was incentivized with 6 CNY. After WJX's initial screening, our team rigorously annotated responses, filtering invalid entries (e.g., nonsensical inputs). finally yielded 1,000 high-quality queries meeting DSL annotation standards, ensuring both diversity and alignment with real-world planning scenarios.
J.3
DATA CONSENT
When collecting the data, we clearly informed the participants about the usage the data and the potential inrevesiile risks of it becoming part of public dataset. We did not track the ID information of the questionnaire respondents. Additionally, we reminded participants not to include any sensitive personal information in the questionnaire responses. During the data cleaning process, we directly removed queries containing offensive content and filtered out sensitive identity information.
J.4 INSTITUTE ETHICS APPROVAL AND RISK MITIGATION
Our questionnaire posed no more than minimal risk: it collected only non-sensitive travel preferences, caused no physical psychological harm, and preserved participant anonymity. The foreseeable risks were limited to minor time cost. All participants were clearly informed about data usage and gave voluntary consent. our institute, minimal-risk studies like ours are exempt from convening dedicated ethics committee. Moreover, our institute explicitly confirms that our questionnaire minimized any potential risk to participants and formally authorized the creation and release of the benchmark.
The risk mitigation strategies we employed are as follows.
Risk Assessment and Disclosure: We conducted thorough assessment concluding the study posed minimal risk. All identified potential risks were fully disclosed to participants. Informed Consent:
Written informed consent obtained from every participant prior to involvement. Consent documents clearly explained the study purpose, procedures, potential risks, data handling (anonymization and usage), voluntary nature, and right withdraw. Privacy Protection: Striet data
39

---

Published as a conference paper at ICLR 2026
anonymization protocols were applied. No personally identifiable information (PII) is present in the collected or released dataset. Data security measures were enforced. Voluntary Participation and Fairness: Participation was voluntary, and fair compensation was provided. Thank you for your suggestions. We will add them in the final revision.
J.5 RISK STATEMENT FOR PARTICIPANTS
Here's 's the English translation of our risk statement for participants:
This questionnaire aims to create open-environment travel planning dataset support academic research. Important Notes:
Data Ireeoccbility: As public dataset, submitted data may not be revoked once published.
Indefinite Retention: As a public dataset, submitted data may retained indefinitely.
Anonymization: All submitted data will be anonymized.
Sensitive Information: Please DO NOT include any sensitive personal information in your responses. (Note: We will collect limited personal information solely to analyze data source diversity. This information will be strictly protected and used only for this specific purpose).
Dataset License: The dataset will be released under the CC BY-NC-SA 4.0 license. Summary: This license allows free use, modification, and sharing for non-commercial purposes only, provided users:
Give appropriate credit (attribution), Share any adaptations under the same license (share alike), And do not use the material commercially.
Full License: We strongly recommend reviewing the complete CC BY-NC-SA 4.0 license terms: Consent Declaration: By submitting this questionnaire, you explicitly consent to our use of the data you provide for non-commercial purposes, including but not limited to:
Algorithm/model development and optimization. Publication academic research. Any other uses permitted under the CC BY-NC-SA 4.0 license.
J.6 CHARACTERISTICS OF ANNOTATORS
Our data collection process solely involved travel requirements and did not include any protected information, such as sexual orientation or political views as defined under the General Data Protection Regulation (GDPR). All data were collected from native Chinese speakers to ensure that the travel requirements fully align with the context and nuances of the Chinese language. This approach was taken to accurately capture the needs and preferences of the target population, which is primarily composed of Chinese-speaking individuals. The annotators were recruited from a diverse range of academic backgrounds, including undergraduate, master's, and doctoral students, provide broad and representative set of travel requirements.
J.7 DSL ANNOTATION FOR HUMAN DATA
The annotation process for the human data involved four stages to ensure the accuracy and validity of the Domain-Specifie Language (DSL) annotations: (1) Initial DSL Version Generation: GPT-4o was utilized to provide the initial version of the DSL annotations for the human data. This step aimed leverage the language model's capabilities generate a baseline for further refinement. (2) Data Annotation Team Revision: A team of five data annotators was responsible for reviewing and revising the DSL. annotations. The team members divided the workload and made necessary corrections to the DSL. annotations to ensure their accuracy and relevance to the travel requirements. Developer Verification and Correction: Three of the main developers of the benchmark conducted thorough review of all the DSL annotations. They verified the correctness of the annotations and made revisions as needed. This stage also involved the exclusion of any invalid queries that could not be verified within the sandbox environment. (4) Final Verification by Primary Developers: The same three main developers performed a final check on all the DSL annotations. This step ensured that the annotations were accurate, consistent, and met the required standards for the benchmark.
(3) Primary
40

---

Published as a conference paper at ICLR 2026
Throughout the annotation process, the focus ensuring that the DSL annotations accurately captured the travel requirements and were valid within the context of the China Travel benchmark sandbox environment. The annotation process for human data required deep understanding of the ChinaTravel DSL. and involved joint debugging and verification with the sandbox information. This significantly limited the size of the annotation team, only limited number of annotators had the necessary expertise and familiarity with both the DSL and the sandbox environment. Additionally, the process was time-consuming and required meticulous attention to detail, further constraining the rate at which the human dataset could grow. Despite these challenges, the rigorous annotation process ensured the quality and reliability of the human data, which is crucial for the evaluation and development of language agents in real-world travel planning.
J.8 TEMPORAL COVERAGE OF HUMAN DATA
The Human- 154 Val Set was collected from August 2024 to October 2024. The Human-1066 Test Set was collected longitudinally over significant period, spanning November 2024 to April 2025. The six-month collection window for the Human- 1000 test set ensures natural temporal diversity. This longitudinal approach captured broad spectrum of real-world implicit intents related to major festivals (c.g., Spring Festival), seasonal travel patterns (winter breaks, spring outings), and varying weather/peak periods. This confirms our test set not overfitted single season, thus providing a robust evaluation of agent generalization capabilities.
K THE IMPLEMENTATION OF TTG BASELINE
K.1
CONSTRAINTS FORMULATION
TTG (Ju et al. 2024) models the travel planning problem as a MILP (Mixed-Integer Linear Program- ming) problem. We adapt their formulation into ChinaTravel for solver-based optimization and the specific parameters, variable and constraint settings can be found in Tab. 21p2
K.2 EXPERIMENTAL SETTING
Although TTG performs very well on Travelplanner, the solver takes slightly more than I second on average to complete the computation. On the ChinaTravel benchmark, the rapid growth of constraints in TTG becomes computationally prohibitive. constraints will exceed 10B (For detailed calculations of variable sizes and the number of constraints,
use the full sandbox, the average number of
please refer to Tab. 2425). Therefore, we only include 22 POIs (2 hotels, 10 attractions, restaurants, 5 stations, 100 intercity transports each for arrivals and departures) and use one hour as time step. We use LLMs to select them from sandbox to ensure sufficient flcxibility in handling different queries. Nonetheless, its constraint scale still reaches 320k x days and the number of variables also reaches 36k x days. In comparison, the commonly used benchmark for evaluating MILP solvers, MIPLIB Glexnnrr al. (2021), contains only 10 instances with more than 320k constraints and about 60 instances with over 36k variables (out a total of 1065 instances).
In main experiments, using the SCIP solver from the PuLP package, TTG allocated relaxed 15-minute search limitation. However, this configuration yielded only 18% valid solutions on easy-subset instances, with the final pass rate (FPR) further reduced to 8% due the solver's pruning heuristics. Fig. a) illustrates the solution time of TTG on 1- to 3-day itineraries. Within the time limit, solutions were found for merely 23% of two-day and 6% of three-day itineraries.
41

---

Published as a conference paper at ICLR 2026
Parameter
Meaning
hotelNum
Number of hotels
altrNum
Number of attractions
restNum
Number of restaurants
transNum
Number of transport modes
stationNum
Number stations
goNum
Number arriving trains/buses
backNum
Number departing trains/buses
timeStep
Number time steps
locNum hotelNum + attrNum + restNum
Total number of POI locations except stations
totalNum locNum + stationNum
Total number of locations including stations
Table 21: Definition of parameters used in TTG
Variable
Meaning
u <[idx]] [2]
The traveler is at location idx at time
event[f
The traveler's location changes at time
hotel idx [d]
Number of times the traveler visits hotel idx on day (d + 1)
attr attr[idx]
Number of times the traveler visits attraction idx
rest(ice [meal]
Number of times the traveler visits restaurant i dx at meal meal 1
Photels Pattry 2rest;
Auxiliary variables
necdEat[m]
Whether the traveler needs to eat meal (during intercity travel)
check[idx][[]
Whether the attraction idx is open at time
j. tr,t)]
The solution, matrix of shape totalNum totalNum transNum x time
Table 22: Variables used in TTG
42

---

Published as a conference paper at ICLR 2026
ChinaTravel
TravelPlanner
M $2 A. R. 19 S 1400 AEI.. W M
Please help me plan trip from St. Petersburg to Rockford spanning days from March 16th March 18th, 2022. The travel should be planned a single person with budget of 11,70.
Current location: Wuhan. want visit Suzhou for a day by myself with a budget of 1,400 RMB. Please provide
with travel plan. - A K.
2 MA I8
Please design a travel plan departing from Las Vegas und heading to Stockton for 3 days, from March 3rd to March Sth, 2022, for one person, with a budget of $1,400.
Xt fx TH EUR (5) 17 M
Current location: Nanjing. want travel
Chongqing alone for 3 days. like sweet foods and bread. Please provide me with travel plan.
32 &2 1 A 2: FT. A SE. I - REf 18 kil.
Craft a travel plan me to depart from New Orlcans and head to Louisville for 3 days, from March 12th to March 14th, 2022. I will be travelling alone with a budget of $1,900.
K.
Current location: Chongqing. My friend and want visit Wuhan for days and try the local cuisine. Could you please provide us with travel plan? K.
" for F Hi. Ail 1 A 2:
Could you in curating a S-day travel plan for one person beginning in Denver and planning to visit 2 cities in Washington from March 23rd to March 27th, 2022? The budget trip is now sct at $4,200.
An 2 12 MT wnth 4010 fietr
JVR dal.
Current location: Chengdu. The ihree want to
visit Shenzhen for days and are interested in historical sites. Could you please provide with travel itinerary?
"A fi fir W.X.I
Could you assist in crafting travel itinerary for 5-day, singlo-person trip departing from Orlando and touring 2 cities in Texas? The travel dates should range from March 10th to March 14th, 2022, entire travel budget $3,100.
WI FR11 1 100
Current location: Shenzhen. My friend and want visit Shanghai for days and we would like to go to the Ocean Aquarium. Could you please provide with travel plan?
S TI HT.
bi
Could you help me arrange a 7-day solo travel itinerary from Kona to California with budget of $5,800, intending visit distinct cities in Califoria from March 7th to March 13th, 2022?
X.
t
AT DE
JF BO
ti iM
bll.
Current location: Chengdu. My friend and I want to visit Shanghai for 3 days. We need twin room, and we might need a meeting space during our stay. Please provide me with travel plan.
H P 22 AI + AE HB
Please help craft 7-day travel plan, I'm planning on leaving from Punta Gorda and exploring 3 different cities in Wisconsin from March 16th to March 22nd, 2022, The budget for this trip is set at $5,700.
am currently Nanjing and plan to travel to Shanghai with two friends for two days. We have chosen the YuanShe Zai Shui Yi Fang Resort Hotel. Please help us plan travel itincrary.
(M R.. #1 - A BE & EAR $11.7 X. 1 fr. Th WE in
Could you help me create 7-day travel plan starting on March 18th, 2022, and ending on March 24th, 2022? The trip will start in Washington and would like to visit cities in Minnesota. This trip is for one person with budget of $7,200.
Current location: Beijing. My three friends and are planning to visit Chengdu for two days. have chosen to travel by train and use subway for city transportation. Please provide me with travel itinerary.
Figure 13: Examples of easy-level queries from China and TraveIPlanner.
43

---

Published as a conference paper at ICLR 2026
ChinaTravel 22 A 2:055155 R. 10
TravelPlanner
Could you please arrunge a 3-day trip for two, starting in Sacramento and heading to Atlanta, from March 14th March 16th, 2022. The budget for this trip is $4, 407, and we require accommodations where parties allowed.
4000 A
# it AE
#*
1 M
fi AN
Rll.
Current location: Wuhan. Two of us want visit Suzhou for 2 days with budget 4000 RMB. We plan to take the train and stay in room with king- size bed. We would like visit natural attractions like Tiger Hill Socnic Arca. Please provide travel itinerary.
ai 100 für MH RE 1 3 K. BVW 9000
Could you please design 3-day travel plan
A 4 MIRR 18330
group of 5, departing from Manchester and heading to Charlottc, from March 29th to March 31 2022? Our budgel is set at $4,800 and we would prefer to have entire rooms for our accommodations.
1 ME RI.
Current location: Guangzhou. Two of us want to visit Chengdu for 3 days with a budget of 9,000 RMB. We plan travel round-trip by train and stay room with a double bed. Could you please provide travel itinerary for us?
10 I" M4. #e PO 1 HB 22 te de bilby X.
Could you tailor 5-day travel plan for two people, departing from Knoxville and visiting cities in Florida from March 20 Murch 24, 2022? Our budget is set $3,900. We'd love explore local Chinese and Mediterranean cuisines during our stay.
100%% 2100 AE (E A
Mi. BIG to DE bi. Current
location: Guangzhou. My two friends want go to Shenzhen for two days. Our budget is 2, 100 RMB. We plan stay in two twin-bed rooms, travel around by metro, eal seafood, and visit Shenzhen Happy Valley.
19 for M. R #2
F. B9W
Could you help create 7-day travel plan for group of 3, departing from Greensboro and louring 3 different cities in Georgia from March 10th March 16th, 2022? We have new budget of $4,000 for this trip. We'd also appreciate if our accommodations have smoking areas.
7000 AET As WHILEE fi B 1650
8th
M mul
Current location: Wuhan. Two of us
visit Hangzhou for 3 days with budget of 7,000 RMB.
We plan travel by planc round-trip and stay in room with large bed. Could you please provide a travel plan for us?
"i fot W. BiUH. 4epl f 2 2 R. Mil 3500 A
Could you help create S-day travel itinerary for a group of 4, starting from New York and visiting 2 cities in Louisiana from March 15th to March 19th, 2022? We have budget of $12,300. Please note that we roquire accommodations where smoking is permissible.
# ti A f
WH # R.
Current location: Hangzhou. Two of want to visit Suzhou for 2 days with budget of 3,500 RMB. We would like to stay in room with a large bed and visit garden attractions like the Humble
Administrator's is Garden. Please provide a travel plan. 210 tit IE. 1e A de b5 R. Hill Wiili * -F. W 1
Can you provide me with 5-day travel plan for 2 people, starting from Asheville and exploring 2 cities in New York from March 13th to March 17th, 2022? Our budget is set at $4, $4,700 and would love to try local Mexican and Chinese cuisines during our trip.
7000 A A A
46 H.
A7 ikii.
Current location: Beijing. Two of us want to visit Shenzhen for 3 days with a budget of 7,000 RMB.
We would like to stay hotel with king-size bed and preferably swimming pool. We plan to fly there and would like to visit Shenzhen Happy Valley. Please provide travel itinerary.
Figure 14: Examples of medium--eveel queries from ChinaTravel and TravelPlanner.
44

---

Published as a conference paper at ICLR 2026
ChinaTravel
TravelPlanner
L 10 br Mi A 2, A.ll
Can you create a 5-day itinerary for group of 7 people traveling from Richmond to two cities in Florida between March 9th and 13th, 2022? Our budget $8,500. We require accommodations that allow visitors and should ideally be entire rooms. In regards to dining options, we prefer French, American, Mediterrancan, and Italian cuisines.
R 2 X. 19.3% 1500 (
to (.)
HI ii 78
[Current location: Wuhan, Destination: Nanjing,
Number of travelers: 2, Duration of travel: 4 days] My classmate and I planning visit Nanjing for days. budget 1500 (excluding transportation and accommodation), just for activitics and meals. Please
help plan,
[ to THE RR Bor ME A 5]
Could you help design a travel plan for two people leaving from Houston to Pensacola for 3 days, from March 6th to March 8th, 2022? Our budget set at $1,400 for this trip and we require our accommodations be visitor- friendly. We would like to have options dine at Indian, American, Chinese, and Italian restaurants. We also prefer not to self-drive during the trip.
H. -
ETREE Jel, 4
1% 1
A
19.77 8000 Alt d
A
[Current location: Nanjing, Destination: Chengdu, Number travelers: Travel days: 5] Our family three wants to travel to Chengdu for a week. We mainly want to visit attractions suitable for children, with a budget of 8,000 yuan, and also taste some local delicacies.
[ ROT PU HH, B BY W. BI PT A. A fT
Could you help create 3-day travel plan for two people? We're traveling from West Palm Beach to White Plains, visiting only one city from March 5th to March 7th, 2022, We have budget of $2,600, For our accommodations, we'd like rooms that are not shared. We are not planning on self-driving and will he reliant public transportation. Cuisines we are interested in trying include Mexican, Chinese, Mediterranean, and American.
2] RM SIT--f-- fJ A MU An 2 BJ FN FNU R.
#
( A DJ lie de AN 06 de 1i. E bz
of d
[Current location: Guangzhou, Destination: Shenzhen, Number of travelers: 3, Number of travel days: 2] Our group of three plans to travel from Guangzhou to Shenzhen for two days. We want to explore bustling neighborhoods, minimize inconvenient transportation, and keep the total expenses as low as possible.
[ H0 WT 33 H. H to W. AWA.HKK A 4, Mi &t 2]
Could you generate a 3-day travel plan for a group of people, departing from Bangor and visiting Washington from March 21 March 23rd, 2022? Our budget is set at $3,100. We require accommodations that are pet-friendly and we would prefer to have entire rooms to ourselves. We do not plan on self-driving for this trip
1 A BCH 2
Dj Ha
1 F-
[Current location: Suzhou, Destination: Hangzhou, Number of travelers: Duration of travel: days] would like people to go to Hangzhou for days to explore historical and cultural sites and have some fun along the way.
[
bi It. #
A 3t MAL XC $ -
Could you help with creating 5-day travel plan for people, originating from Evansville and covering 2 citics in Texas from March 17th to March 21st, 2022? Our preferred
3]
Rt Ai
12 #6 (1.
38
[Current location:
Shanghai, Destination: Beijing,
Number of travelers: Number of travel days: I want to depart from Shanghai and spend three days in Beijing. I hope see some famous landmarks and try some local specialtics, with a sufficient budget.
accommodations are private rooms, and they must permit children under 10 since we will have them with us. Transportation should not involve any flights. The budget for this trip set at $2,800.
Can you assist in creating a travel itinerary for a group of starting Seattle and visiting 3 unique citics across Texas? This trip will span over 7 days from March 10th through March 16th, 2022, We have budget $11.000. Regarding our accommdaations, we would like to rent entire rooms, and it's important that lodgings allow partics. As for transportation, we do not plan to drive ourselves around. Figure 15: Examples of human/hard level queries from ChinaTravel and TravelPlanner.
[ Ao it El by W. L. Hi # 2, Ett
3] H A0 X
bi. it
RE M An &I do G1 90.
[Current location: Beijing, Destination: Shanghai, Number travelers: Number of travel days: 3] My friend and I are planning to spend thrce days traveling from Beijing Shanghai. We plan to fly round trip and prefer a red-themed travel route.
45

---

Published as a conference paper at ICLR 2026
Incorrect API Calls
Repetitive Output Loops
Action/20|: "0 start-"Gtll lits" kud=
Thouuahl 121: By IS 10 MO SU29 LL LE 19
time-"19:00", method="'us
Obsrrvatio] 200; Invalid method. Only suppor listroe, "alk", "eadi]]
Actioal allirn selee Melletillly (illyy "IEE aey name""
bunk lambdda da Whisee
Action|25]: yoluteity "XMM"" sta--" JEts", and-r AV".
start Uime-"19:00" method-"'us bus
Thought 131: 12 to Ro
Observation 25: Invalid method. support [imetrot, walk, taxi")
HUM NI WEEES 1 6R.00 Eiiii
Actioon11 attmectioss soloctfetty 103 tramet
Action]30]: godal, start-"elll Illll 112 and--//il KIBA kt",
Fune Lambada uy "Brayre" in 3:29 ti
"1:00"9:007; mothod="bus
Observation [30; Invaled method. Only support [mmerro,, walk", taxi")
Thought 14]:
221 141:22211000000 BE is 3999-- 1n 95 2813 98 12.000 1ESII
Action[14 attmetions solecticity"
hunc-lambda xy 1o inx 3e M.
Action[3]: fity Iansport stlentstart rig=" W"", end "
inlemily bypes" uniin"" eutlest leave lim==119:00")
Though 15 P2T
Observation 3]: (K
Et name 'aity traneport
8B MA TERR38 4 dh Hill M.
not defindii @ it M.BU /2 fijkk..
Actios attaccionn selecticily
Thought[4] 82
Liiitnnmyy transport seloct. 11.00
lunc=lamlda x: "Br Whagee in
(UTI (4:1 (4:0) 39
Reasoning- ction Inconsistency
Critical Information Missing
Thought[2] 58..6 F0 JIF, WRESE HE
Actiou/15): glannectquery- RU Hi. ft A-A-AAAA Al R
TE 6630000l BI.
K. 3000 A With ft (oe----00ti 1-
Action]2]": internity imnsport selectistart Wk3l" end city
11 finished, The
is: p'pooole number:: 'start city": HH* "target city;; "iix"" "itneerry | day"
i8 intumily agee sirplan"))
Observation2]: Righhtl) "From" "To" "BeginTime 'EndTima"
"activities ("start lime"; "auu"""188:" 118.00 "end time:: "22:3' "stat":
'Duration" 'Con' [V("F12340 VBUUU BUA-
Mil
aUScR!" "end': Fitts))" "cost": 543.6, "TrainlD" "087". "type": "Inin" "Gckets" : 1, "Iaasporrt"" UОНН
BUS
14 BUREL R Bi SETT Ut tu
Action plannerqqueryy 101t W.lH A AE R. A. Wit 11 12 "y"
Bl.
Action[14] plannerqqurry " H."". J...--RR JURRU K
Observation[23) Task finished The mower ("people number': "3i4"" "Mimenny" lrdda"
ix. 09313100 A LET., With A fe
"Bell" "tarpt
Observation| [4]: Task finshhed. The answer is! ('popple namber's 2,
"activitie" [["ope": "tran", "stat time" "06:33" "end time":
"statt city:: "I"""" "baged city"
"ilmmrar" d'd
"15:46", "coa"" "o 739.35, "ickets "sta": "Wotll"" "end", 339H "TrainDD "G278"" "transports" ": (D) .....
ardivibess: [["start time' 708:00", "ond limet: 14.48", shart
"Ill
H HIMAS end" 11W0 "oost: 787.73, RbgghDD" "G818" "tmnsports". [I], ("pasition:: "698", "type" "attractoon, "uraspprtts: [[ (('star": (.A. 'end": tat Hf Ai-BBbidd. "mode": "wal""
R &1 g",, "end "5H00F, "trampott" (]), ('tppe: "tmin", "slart lime"; "22.33" "end, lime"; 107.00 "cuat"; 759.35 "ückets":
"ype"
"start" 106", "18A00"" "Tminl "G2788", "trnspocts" (I))))
Figure 16: Fail case studies of React-one-shot DeepSeek Method.
46

---

Published as a conference paper at ICLR 2026
1. 22
CAN W. L. BEMI, RAM. atAN ITA, NEM. JERE 3544-113
BUH
BUHI, BeII, ZER. HREE WIK. #5
2. HO
AF
(1-5)
4. A
(1-5)
18: 1 STTT:
FIRT, 18 BA ta A
St. サ 18
,
7P
# & AME
19192 3000 A
7:
:
B :
: WOUON
05:24>08::15
:22.
:
2
(
-WO Bid))
+1 23
8 3.
:
:
- 188
10:11010 1>13:10
# 169
::370 N
: 355433
#:67220, 20:09-221:23
1222
12
15 2 Add HE
4 - B AEEE
# Se
& AL ME # 3
Figure 17: Questionnaire
47

---

Published as a conference paper at ICLR 2026
Open Travel Planning Data Collection Questionnaire
This questionmuire aims constract dataset for travel planning in open environment to facilitate relevant research. Since the responses will be part of public dataset and cannot be revoked, please do not include any sensitive personal information in your respones. Thank you your participation!
Departure City:
(Choose from Beijing, Nanjing, Shanghai, Hangzhou, Shenzhen, Wuhan, Guangzhou, Chengdu, Chongqing.
Suzhou)
2. Destination City:
(Choose from Beijing, Nanjing, Shanghi, Hangshou, Shenzhen, Wuham, Guamgzhou, Chengdu,
Chongging, Suzhou)
3. I Number of Travelees:
(1-5)
Number Travel Days:
(1-5)
As user, can submit queries to the intelligent agent. Your query may include specific requirements for attractions, dining, acommedation, intercity transportation (e.g.. train, planc), and intra-city transportation (c.g. subway, walking. taxi). may also provide personal preferences. Please number of travelers, and the number of travel days, and make sure they are consistent. The intelligent agent will generate travel plan based on your roquest, covering transportation arrangements, accommodation, recommended attractions, and dining suggestions.
that your query includes the following three pieces information: the destination the
Example User Query:
"My current location Suzhou. I want to travel alone to Nanjing for days with budget of 3,000 RMB, taking the high-spood train for both departure and return. Please provide travel plan."
Examplc Response from the Intelligent Agent:
Departure: Suzhou
Destination: Nanjing
Transportation: Suzhou North Station -+ Nanjing South Station
Train; G4, 07:24 08:15
Cost: 122.9 RMB
Tickets:
Attraction: Xuanwu Lake Scenie
Transportation: Subway (Nanjing South Station --
Nanjing Forestry University Xinzhuang)
Route: Walk minutes
Subway 23 minutes
Walk minutes
Cost: RMB
Visit Time: 08:50 -+ 10:00
Admission:
Nanjing Jinling
Chinese
Cost: 188 RMB
Time: 12:10 13:10
Accommodation: Crystal Orange Hotel Nanjing Xuanwu I Lake
Room Type: Queen Room, room
Retum: Nanjing South Station -> Suzbou Station
Train: G7220, 20:09 21:23
Cost: 122.9 RMB
Tickets:
Classification of User Queries by Difficulty I. Level
We categorize user queties into difforent difficulty lovels as follows:
Easy Level: General inquirics without personalized requirements.
Medium Level: Includes some dogree of personalizaton, usually involving food, lodging, or transportation.
Hard Level: Involves
complex and specific needs, such lime consiraints, particular locations, planned activities.
Examples User Queries at Different Difficulty evels:
know the itinerary for 2-day trip Shanghai from Hangzhou." Internediate Level: "I plan to travel alone Nanjing on budget and stay for about three days. I'm interested in history and culture and would explore historical sites depth."
Basic Level: "I
Advanced Level: "Three of need travel to Beijing the day after tomorrow for 2-day trip. We need return from Beijing Railway Station before PM on second day. We want to visit the Forbidden City on the first day and Temple Heaven on the second day, Please provide travel plan." .
Please provide user query:
Figure 18: The translated version of the questionnaire
48

---

Published as a conference paper at ICLR 2026
Prompts for POI recommendation
NEXT_ TET_LOO__YYEE_IITTUUUI
You are travel planning assistant. The user's requirements are: ().
Current travel plans are: ().
Today is (), current time is (h, current location is (), and POI_type_ine list is (}.
Select the next POI type based on the user's needs and the current itinerary.
Please answer in the following format.
Thought: [Your reason]
Type: [type in Pol_type_list]
Figure 19; Prompts for next-POl-type recommendation
49

---

Published as a conference paper at ICLR 2026
Prompts for restaurants recommendation
RESTAURANT RANKING INSTRUCTION .un
You are travel planning assistant.
The user's requi are: (user_requi rement.s). The restaurant info is: (restaurantjinfo)
The past cost for intercity transportation and hotel accommodations is: (past_cost).
Your task to select and rank restaurants based on the user's needs and the provided restaurant information. Consider the following factors:
1. Restaurant name
2. Cuisine type
3. Price range
4. Recommended food
Additionally, keep in mind that the user's s budget is allocated across multiple expenses, including intercity transportation and hotel accommodations. Ensure that the restaurant recommendations fit within the remaining budget constraints after accounting for the past cost.
Note that the price range provided for each restaurant is the average cost per person per meal, the remaining budget must cover the cost of three meals per day for (days) days.
For each day, recommend at least 6 restaurants, combining restaurants for all days together.
Your response should follow this format:
Thought: [Your reasoning ranking the restaurants] Restauraat.Nmmeliss:: [List of restaurant names ranked by preference, formatted as a Python list]
Figure 20: Prompts for restaurant recommendation
50

---

Published as a conference paper at ICLR 2026
Prompts for attractions recommendation
COTTACTIONLRAKKNGGINNN
wun
You are travel planning assistant.
The user's requirements are: riser_requremmetss) The attraction info is:
(attractior_info)
The past cost for intercity transportation and hotel accommodations is: (past_cost).
Your task is to select and rank attractions based on the user's needs and the provided attraction information. Consider the follewing factors:
1. Attraction name
2. Attraction type
3. Location
4. Recommended duration
Additionally, keep in mind that the user' budget is allocated across multiple expenses, including intercity transportation and hotel accommodations. Ensure that the attraction zecommendations fit within the remaining budget constraints after accounting for the past cost.
For each day, recommend at least 8 attractions, combining attractions for all days together. To ensure comprehensive list, consider larger pool of candidates and pricritize diversity in attraction type and location.
Your response should follow this format:
Thought: [Your reasoning for ranking the attractions] AttractionNameList: [List of attraction names ranked by preference, formatted as Python list]
Example:
Thought: Based on the user's s preference for historical sites and natural attractions, the attractions are ranked as follows:
AttractionNameList: ("Attractioni", "Attraction2", ...]
Figure 21: Prompts for attraction recommendation
51

---

Published as a conference paper at ICLR 2026
Constraint Type
Mathematical Formulation
Spatio-temporal
[[idx][[] [[dd][[ 1] u [idx][4]
Constraints
8[idx] [t] 2 u [idx] [t] 2 [dx [ + 1]
event[t] = 0 + [[idx][t] u [iddx][ + 1]
event[t] + Ena d[idx][d] 2
E; u//i[tt] =
Hotel Constraints
Photel [idx][] = u [iliix[ [4] A event(4)
hotel [idx][]]] = Σ
(d+1) stepPerDay
Zhotel [idx] [t]
t=d Stap
Eiex hotel[idx]]d [c]]
Attraction Constraints
Zatr [idx[(]] 2 [e[idx] [t] A event(t)
attr[idx] E. Zatr jidx[4
Eua attr[idx] 2 min attr
[ix]][idx][tt False uiiix][tt =
Meal Necessity
needEat[m) = 1 + a(ma < Tasp
needEat[m] = 1 = b[m] > Tar
Innercity Transport
v((i, (...j.tram tran, r)] < u [1]]
Constraints
v((i, J, tran, 6)] < event[t]
vl(i. j. tran, t)] S u u[tran] [++ 1]
v(( j, tran, t)] < u/tran| [t+ 6]
vl(i, j, tran, t)] S event(t + 6]
v((i, j. tran, t)] _ u|ii[( + 5 1]
Restaurant Constraints
res [idx] [t] 2/ [diix]] [t] event(t]
rest [ix [m]] = _ Frest [i]
Eiar rest(idx][mm] s needFat[m]
check [idx = False = u[idx][t] = 0
Intercity Travel Constraints
Σ, interGo[i]
Σ, interBack(]
interGo[i] = + 2s IgoStation
[+]
interBack[
a/backStation [
Table 23: Constraints used in TTG
52

---

Published as a conference paper at ICLR 2026
Variable
Dimension
[idx] [t]
(totalNum + transNum) timeStep
[[idx][t]
(totalNum + -transNum) x (timeStep - 1)
event[t]
timeStep
hotel[idx]]
hotelNum x days
Photel [td
hotelNum x timeStep
attr(ide attr[idx]
attrNum
Zuur [idx]] [t]]
attrNum timeStep
rest[idx] [idx] [meal]
restNum X 3 x days
Frest [d]][[]
restNum x timeStep
Ji. tr, t)]
totalNum x totalNum x transNum x timeStep
total
days x stepPerHour x 36k
Table 24: Variable sizes in TTG
Category
Estimated Size
Spatio-temporal constraints
(totalNum + transNum) x (4 timeStep + 3)
Hotel constraints
hotelNum (3 timeStep days)
Attraction constraints
attrNum timeStep
Restaurant constraints
restNum (4 timeStep days)
Urban transport constraints
7 totalNumi
x transNum timeStep + totalNum timeStep
Intercity transport constraints
(goNum + backNum) timeStep
Table 25: Number constraints sizes TTG
53


Reviews:
[{"review_id": "vevgCTGRQo", "reviewer": "Reviewer_nBDf", "summary": "This paper introduces ChinaTravel, a new benchmark for evaluating language agents in complex, real-world travel planning. The authors argue that existing benchmarks are overly synthetic and lack realistic, open-ended human queries. ChinaTravel contributes: (1) a sandbox environment for multi-day, multi-POI planning in Chinese cities; (2) a domain-specific language (DSL) for compositional constraint specification and automated validation ; and (3) an open-ended dataset derived from 1154 human participants, featuring implicit intent and novel constraint compositions. Experiments show that pure LLM agents completely fail , and while neuro-symbolic methods perform better, significant challenges in contextual grounding and compositional generalization remain.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The benchmark is novel and well-motivated. The paper clearly articulates the limitations of prior benchmarks like TravelPlanner. \n\n- The benchmark includes 154 human-validated queries and 1,000 survey-collected queries reflecting real-world travel requirements with implicit expressions \n\n- The experiments convincingly show that ChinaTravel is a challenging benchmark. Pure LLM methods (ReAct, Act-only) completely fail, achieving near-zero Environmental Pass Rates (EPR) and Final Pass Rates (FPR). This confirms the task is beyond the reach of current text-wise planning approaches.", "weaknesses": "- It is unclear how much of the performance gain (e.g., 37.0% C-LPR in Table 3)  comes from the (1) iterative NL2DSL translation, (2) the symbolic search sketch, or (3) the LLM-driven POI ranking within the search. The ablation study in Sec 4.3 only explores preference ranking (PEQ vs. PDS) and doesn't dissect the core \"NeSy Planning\" search algorithm itself.\n\n- Inter-annotator agreement is not reported for DSL annotation process. With five annotators performing initial revision and three developers conducting verification, consistency metrics (Cohen's kappa, Fleiss' kappa) would strengthen quality claims.\n\n- Temporal coverage is unclear: Were queries collected during specific seasons? Travel requirements vary seasonally (festivals, weather, peak/off-peak periods), but dataset doesn't indicate collection timeframe or seasonal distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uoKUlOdPMD", "reviewer": "Reviewer_DfjZ", "summary": "- The paper introduces ChinaTravel, a realistic Chinese-language benchmark for multi-day, multi-POI travel planning grounded in real human use cases.\n\n- It provides a compositional DSL and neuro-symbolic framework to express and verify logical and preference-based travel constraints.\n\n- The benchmark enables evaluation of LLMs and hybrid reasoning models across feasibility, constraint satisfaction, and preference comparison.", "questions": "- (minor) The citation style at L 41 needs to be corrected.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper builds a realistic and complex dataset grounded in real human travel use cases, making it well aligned with real-world scenarios.\n\n- The authors provide a Chinese-language benchmark, offering a valuable resource beyond existing English-centric datasets.\n\n- The benchmark enables multi-constraint, compositional planning that captures the true complexity of real travel tasks.\n\n- The study offers comprehensive evaluation with diverse models, baselines, and metrics.", "weaknesses": "**W1. Language diversity**\n\n* The dataset is **Chinese-only**. While introducing a non-English benchmark is valuable, including English or multilingual settings would improve usability and comparability with other benchmarks.\n\n**W2. Limited novelty**\n\n* Although the dataset moves closer to real-world settings, the contribution feels **incremental** — mainly increasing constraint complexity within existing benchmark scopes, without introducing fundamentally new ideas or designs.\n\n**W3. Lack of comparison with existing benchmarks**\n\n* While the paper claims that LLMs struggle in ChinaTravel, it does not compare performance on prior datasets (e.g., TravelPlanner, TripPlanning).\nAlthough direct comparison is difficult due to language differences, the authors could still replicate the settings of existing benchmarks to provide a contextual baseline.\n  Such analysis would strengthen the benchmark’s validity.\n\n**W4. Limited evaluation depth**\n\n* The evaluation section lacks a detailed analysis of *why* models fail.\n  Studying factors such as the **number and composition of constraints** or their combinations could provide deeper insights into agent behavior.\n\n**W5. Preference metric design**\n\n* The current preference evaluation (e.g., Fig. 8) treats preferences as independent objectives.\n  In real-world scenarios, multiple preferences coexist and may conflict with hard constraints.\n  Therefore, a more holistic metric would be considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "reKZ9sj4F8", "reviewer": "Reviewer_75FN", "summary": "This paper introduces ChinaTravel, an open-ended multi-day travel planning benchmark designed for evaluating language agents under real-world, compositional constraint scenarios.  It introduces a domain-specific language (DSL) for formalizing diverse logical and preference-based requirements, alongside a large-scale dataset combining human-authored and LLM-generated queries. Extensive experiments highlight the advantages of neuro-symbolic methods in constraint satisfaction and preference reasoning over pure LLM baselines. However, the authors also mention that challenges persist in semantic grounding and compositional generalization, where even state-of-the-art models like GPT-4o and DeepSeek-V3 show limited performance.", "questions": "n/a", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The proposed ChinaTravel benchmark is solid and realistic, encompassing diverse travel scenarios, multi-day itineraries, and compositional constraints that closely align with real-world planning needs. It effectively addresses several limitations of previous benchmarks.\n2. The experiments are comprehensive and well-designed, covering a wide range of models, evaluation metrics, and analytical perspectives.\n3. The paper not only introduces a new benchmark but also proposes an integrated methodology that combines large language models with traditional neuro-symbolic reasoning, pointing toward a promising direction for advancing constraint-aware planning.\n4. The analysis is detailed and insightful, offering in-depth examinations of model behavior, constraint satisfaction, and preference reasoning.\n\nOverall, this work presents a well-rounded and impactful contribution, bridging benchmark construction, methodological innovation, and empirical understanding in the field of language-agent-based planning.", "weaknesses": "1. This work is solid and well-executed, featuring a carefully designed benchmark, strong experimental methodology, and comprehensive analysis. However, since it largely builds upon prior innovations rather than introducing a paradigm shift for the field, I cannot assign it the highest rating.\n2. While the authors mention introducing up to 12 constraints per query, I wonder whether real users would naturally provide such detailed inputs when first interacting with a “travel assistant.” In most real-world scenarios, users tend to refine their requirements through multi-turn interactions, and incorporating such a dialogue-based setting could make the benchmark even more realistic and insightful.\n3. Another concern is that the authors did not include large reasoning models in their experiments, which are expected to outperform general-purpose LLMs on complex planning and constraint-satisfaction tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vgbI0AopPt", "reviewer": "Reviewer_a2uL", "summary": "The paper presents ChinaTravel, a benchmark for realistic multi-day travel itinerary planning with open-ended user requirements and compositional constraints. It introduces a domain-specific language (DSL) for formally encoding user constraints and preferences, enabling automatic validation of itinerary feasibility. The benchmark includes real travel data for 10 Chinese cities and 1,154 authentic human queries annotated with executable DSL code. Experiments show that pure LLM-based agents perform poorly on complex constraint satisfaction, while neuro-symbolic methods that combine LLM interpretation with structured planning achieve higher success. Despite these gains, overall performance remains low, indicating the benchmark’s difficulty and the need for more robust compositional reasoning in travel planning agents.", "questions": "- The DSL-based evaluation is powerful, but it relies on a fixed library of concept functions. How easily can this DSL be extended to accommodate new types of constraints or domains? For example, if a user asks for a constraint involving a notion not currently encoded, say, “scenic rating of a route” or “avoid areas with high COVID-19 cases”, would adding such a constraint simply be a matter of defining a new attribute function in the DSL and updating the database?\n - Have the authors considered training a dedicated model for this *NL2DSL translation* task? For instance, using the large set of synthesized queries (Stage II) with their DSL annotations as training data to fine-tune a transformer that directly outputs DSL code could potentially improve accuracy and consistency. It could also avoid the iterative Reflexion loop that sometimes prunes constraints. If this was attempted or ruled out, could the authors elaborate on the challenges?\n - How exactly are soft preferences evaluated in ChinaTravel’s scoring?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Addresses key gaps in prior travel-planning benchmarks by focusing on realistic and multi-day intra-city trips with authentic human queries. Covers diverse implicit constraints and overcomes prior limitations like synthetic prompts and fixed rules. Strongly establishes novelty and real-world relevance.\n- Introduces a well-designed Python-like DSL that enables expressive and composable user constraints. Allows logical, arithmetic, and relational conditions to be defined and verified automatically.\n- Defines rigorous metrics and ensures queries have feasible solutions. The C-LPR metric and per-query vs. per-constraint evaluation demonstrate careful empirical design. Methodology ensures credible and fine-grained performance assessment.\n- Benchmarks a wide range of methods across multiple models. Results show NeSy methods outperform others. Thorough failure analysis adds diagnostic insight.", "weaknesses": "- A potential concern is that the benchmark is tied to China-specific travel data. Agents might overfit to domain idiosyncrasies or to the Chinese language expressions of requirements. The authors do mitigate this by providing English translations of all information for international researchers, but it’s not fully clear how an English-only LLM would perform if the underlying database is Chinese. Moreover, the DSL’s set of primitive concepts is tailored to travel. It’s not obvious how easily this framework would extend to other planning domains, say, event scheduling or travel in countries without the same data availability. The reliance on a predefined set of attributes means truly novel user constraints outside those attributes still cannot be captured without augmenting the DSL, e.g. “scenic beauty of train route” which is not a stored attribute.\n- The neuro-symbolic pipeline is effective but complex and brittle, with multiple interdependent stages prone to cascading failures. Each stage can introduce failure points: e.g., an error in DSL translation can completely derail the planner. Indeed, the results show a sizable drop from an “oracle” DSL setting to the normal setting, indicating the pipeline’s success is heavily dependent on the NL-to-DSL conversion quality. It means the proposed solution might not scale gracefully to even more complex queries or to smaller and less capable LLMs. In essence, the current neuro-symbolic method feels like a very involved engineered solution.\n- The use of micro vs. macro scores for five different rates resulted in dense tables that were a bit hard to parse on first read. For instance, Table 2 presents many numbers for each method, and it wasn’t immediately obvious to a reader what “macro LPR = 0” signifies without reading the fine print. The paper might benefit from a brief explanation in the text of how micro vs macro are calculated (the appendix defines it, but a sentence in main text would help guide the reader).\n- Additionally, the treatment of soft preferences in the evaluation is somewhat under-emphasized. Preferences are encoded as optimization objectives in the DSL, but the main results focus on binary pass/fail of constraints. It appears that preference satisfaction was analyzed separately rather than as part of the overall success metric. This separation makes sense since preferences aren’t hard requirements, but it leaves open questions: for example, if one method produces a plan that satisfies constraints but with suboptimal preferences, e.g., it visits fewer attractions than possible, how is that reflected?\n- Planning remains computationally intensive; even top methods require multi-minute searches and repeated LLM calls. MILP and large-context approaches become intractable for complex itineraries. The paper could discuss strategies to handle combinatorial scaling and efficiency improvements more explicitly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s3zdmO4WxY", "reviewer": "Reviewer_ynKP", "summary": "This paper constructs a dataset for the travel planning problem to address the limitations of existing benchmarks, which typically rely on synthetic queries with limited constraints and explicit intent, diverging from real-world scenarios.", "questions": "Check the weaknesses section for more details.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and clearly communicates its key contributions.\n2. The constructed dataset is comprehensive and appears to be thoughtfully designed.", "weaknesses": "I do not have significant concerns about the technical aspects of the paper. The following are not weaknesses, but some thoughts shared with the authors. \n\nTravel planning is inherently complex, making it genuinely useful for people involves challenges beyond optimization or scheduling. If we view travel planning purely as a multi-day, multi-point-of-interest (POI) planning problem, as the paper does, the formulation seems reasonable. However, from a broader perspective, I am not convinced that the proposed dataset truly reflects how people make travel decisions, or that strong model performance on this dataset necessarily implies alignment with real user preferences.\n\nIn practice, travel planning goes far beyond cost and time optimization. Human preferences are deeply subjective and influenced by social and contextual factors. For example, people often seek advice from friends who have visited a destination, or they explore social media (e.g., Instagram, TikTok) to find inspiration from influencers or acquaintances. These behaviors are difficult to capture in a dataset that models travel purely as a structured multi-step planning task.\n\nTherefore, I recommend that the authors consider reframing their work not strictly as a travel planning problem, but more generally as a multi-step or long-horizon planning task. This broader framing could make the work more generalizable and align it with recent efforts in benchmarking agentic reasoning and planning capabilities. The following papers might be relevant references (I am not the author of any paper that I listed):\n\nReferences\n\n[1] WideSearch: Benchmarking Agentic Broad Info-Seeking\n\n[2] BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5xBy98IWbX", "reviewer": "Reviewer_MJSh", "summary": "This paper presents ChinaTravel, a new benchmark that addresses a critical and increasingly obvious gap in language agent research.\nThey argue that current benchmarks adopt synthetic queries and have limited constraints.\nThus, they provide a more diverse-formatted and a complicated sandbox.\nThey find that near-symbolic planning perform better than the LLM agents.", "questions": "1. The paper identifies that the primary bottleneck in the NeSy is the initial NL to DSL translation. Is there any solution for such problem given you are using existing neuro-symbolic methods?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The benchmark extends the diversity and complexity of existing benchmarks.\n2. The paper adopts DSL to evaluate the outputs of the LLMs.", "weaknesses": "1. The conclusion that the symbolic methods perform better than the LLM agents is obvious. I think the reason of proposing such benchmarks should be testing LLM agents to plan with constraints. We can definitely foresee a greedy search could complete such tasks. Yet, with such external help, the performance of LLM agent itself does not improve.\n2. The DSL is a powerful contribution for validating compositional logical constraints. However, real-world travel planning is filled with subjective, non-functional requirements that are difficult to verify, for example, the user may ask for a chill journey.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yq3bq4Ardd", "reviewer": "Reviewer_bntG", "summary": "This paper introduces ChinaTravel, a new benchmark designed to evaluate language agents on realistic, multi-day, multi-POI (point-of-interest) travel planning tasks within Chinese cities. The benchmark integrates a realistic sandbox environment, a domain-specific language for compositional constraint validation. The authors further explore Neuro-Symbolic planning methods, combining LLM-based understanding with symbolic reasoning for constraint satisfaction. Empirical studies show that NeSy agents outperform pure LLM-based methods in meeting logical and environmental constraints.", "questions": "1. For GPT-4o, why does the NeSy Planning with Oracle Translation method lead to a significant performance decrease compared to the vanilla NeSy Planning approach?\n2. Would the same DSL and sandbox design generalize to non-Chinese or multilingual settings?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes a domain-specific language that programmatically composes atomic concepts of travel attributes.\n2. This paper makes a meaningful attempt to evaluate Neuro-Symbolic Agents for real-world travel planning tasks, illustrating the potential.", "weaknesses": "1. The claimed contributions concerning the Sandbox and Open-Ended Travel Dataset are not sufficiently distinct from existing approaches. The work does not clearly demonstrate a research gap relative to previous studies.\n2. The authors propose a contextual grounding task to capture implicit user intent. While this is both practical and interesting, the benchmark primarily serves to identify such phenomena rather than offering effective solutions or promising directions to address them. It would be more insightful if the paper included a deeper analysis or exploration of potential approaches to tackle this issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)