Towards Full Delegation: Designing Ideal Agentic Behaviors for Travel Planning

Authors: Song Jiang, Da JU, Andrew Cohen, Sasha Mitts, Aaron Foss, Justine T Kao, Xian Li, Yuandong Tian
Primary area: foundation or frontier models, including LLMs
Keywords: Agentic Behavior, LLMs, LLM-as-a-Judge, Synthetic Data

Abstract:
How are LLM-based agents used in the future? While many of the existing work on agents has focused on improving the performance of a specific family of objective and challenging tasks, in this work, we take a different perspective by thinking about full delegation: agents take over humans’ routine decision-making processes and are trusted by humans to find solutions that fit people’s personalized needs and are adaptive to ever-changing context. In order to achieve such a goal, the behavior of the agents, i.e., agentic behaviors, should be evaluated not only on their achievements (i.e., outcome evaluation), but also how they achieved that (i.e., procedure evaluation). For this, we propose APEC Agent Constitution, a list of criteria that an agent should follow for good agentic behaviors, including Accuracy, Proactivity, Efficiency and Credibility. To verify whether APEC aligns with human preferences, we develop APEC-Travel, a travel planning agent that proactively extracts hidden personalized needs via multi-round dialog with travelers. APEC-Travel is constructed purely from synthetic data generated by Llama3.1-405B-Instruct with a diverse set of travelers’ persona to simulate rich distribution of dialogs. Iteratively fine-tuned to follow APEC Agent Constitution, APEC-Travel surpasses baselines by 20.7% on rule based metrics and 9.1% on LLM-as-a-Judge scores across the constitution axes.

Decision: reject
Reviewer scores: 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1730087216705

thread_type:
official_review

<thread_object>
creator: reviewer_utgm

<content>
summary:
The paper proposes a framework, APEC, defining criteria for good agentic behavior termed the "Agent Constitution." APEC-Travel is introduced as a running scenario, demonstrating the framework's applicability to travel planning, with agents trained using DPO to align with APEC principles. The work utilizes rule-based metrics and agentic scores for training, and synthetic data generation creates 54 personas informed by a human survey for personalization in travel planning.

strengths:
1. **Novelty** : The concept of an "Agent Constitution" is compelling and could serve as a foundation for designing ethical and efficient agent behaviors across various applications.
2. **Empirical Validation (Quality)**: APEC-Travel demonstrates the framework’s applicability and effectiveness in generating coherent travel plans, which is evaluated through agentic scores and rule-based metrics.
3. **Personalization Significance**: The introduction of 54 empirically supported personas is a thoughtful addition to enhancing travel planning experiences.
4. **Clarity**: I find few minor details missing with gap in definition and implementation of the metrics. Although the text is well written to explain the major claims made.

weaknesses:
The paper provides an interesting perspective on establishing agentic principles and has shown the potential for real-world impact, especially in travel planning. Addressing the points mentioned could enhance its clarity, rigor, and applicability to broader domains.

1. Validation of Synthetic Personas
    * Clarification Request: The paper lacks clarity on whether the simulated personas were validated beyond their construction from survey results. Given the random entry assignment for synthetic traveler generation, it would be valuable to know if validation methods (e.g., similarity scoring to real user behavior) were employed or considered.
2. Switch from Large to Smaller Model During “Critical Persona Entry Selection”
    * Justification Needed: During Seed dialogue generation and agentic scoring, a larger model (Llama3.1-405B-Instruct) was used, likely for accuracy. However, during the "Critical Persona Entry Selection," a smaller version (Llama3.1-8B-Instruct) was utilized. Is there a performance or computational rationale for this model switch, and how does it affect outcome consistency?
3. Efficiency Definition and Mapping to Experimental Context
    * Clarification Request: The definition of efficiency as "accuracy gain per round" does not seem to align with the context described before the experiments, where it mentions "checking whether the agent has obtained critical information with limited dialog exchanges." Was efficiency calculated by invoking the Stenographer after each round to validate information retrieval or solely by accuracy differential per round? Additional clarification on this could help contextualize the metric’s relevance.
4. Exclusion of Efficiency in Reward Calculation
    * Inquiry: Why was efficiency, a key metric, excluded from the reward calculation for DPO training, despite its apparent relevance in performance benchmarking? Is it because its derived from Accuracy?
5. Sentence Construction in Section 4.3
    * Suggestion: Minor improvements in sentence structure are needed just before Section 4.3. For example, "suggesting the efficacy of our training recipe in consistently train agents that align with the principles..." could be improved.
6. Alpha Value in Reward Calculation
    * Clarification Request: The value of alpha in the reward calculation appears significant, yet it's unclear what alpha was set to for the primary results shown in Table 2. This information would aid reproducibility and could be noted in the table or text if it was overlooked.
7. Alpha Setting in Iterative Training Paradigm
    * Inquiry: In the ablation study, "Iterative Training Paradigm" shows only accuracy improvements between fixed and recursive paradigms. Was this improvement due to a high alpha value in the reward calculation? Further details would clarify the choice and impact of alpha in iterative training.
8. Evaluation Across Domains and Real-World Datasets
    * Suggestion: While APEC demonstrates effectiveness in APEC-Travel, additional evaluation on other agentic domains (e.g., web exploration, embodied agent tasks) would strengthen claims of generality. Additionally, employing established travel planning datasets (e.g., TravelPlanner[Xie et al. 2024b]) could enhance validation beyond synthetic data. I assume this as a limitation of this work.

questions:
I have listed the questions, concerns and points required for clairty above and I am looking forward to the rebuttal during the disscussion phase.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730216779156

thread_type:
official_review

<thread_object>
creator: reviewer_czwh

<content>
summary:
The paper proposes a "constitution" for evaluating the behavior of LLM agents beyond accuracy. In particular, it proposes four criteria: (1) accuracy, (2) proactivity, (3) efficiency, and (4) credibility. In a travel planning setting, the paper then uses prompting-based methods to score agent trajectories on these four categories, resulting in an "accuracy score" and an "agent score" which are combined to form a reward measure. The paper then shows a proof-of-concept that using SFT on data from a stronger model, as well as DPO, scores on this reward can be increased substantially.

strengths:
1. I think the general idea of evaluating agents beyond accuracy is reasonable and well-motivated. There are many aspects of what makes a good user experience (e.g., did the agent provide compelling evidence for its decisions?) that aren't necessarily reflected in task success metrics

2. The experiments are thorough, easy to follow, and include several ablations. Additionally, the approach to persona construction seems reasonable and relatively well-aligned to the preferences of travelers in the real world

weaknesses:
1. The proposed "constitution" lacks a clear motivation for why it chooses the four principles (accuracy, proactivity, efficiency, credibility) that it does. In particular, it's unclear to me why you need all four of these principles: can an agent be accurate and efficient without also being proactive and credible? Additionally, the main experiments in the paper use a different set of principles (plan & priority, proactive, clarification), calling into question the utility of the original set of four

2. The method seems to be primarily focused on distilling the behavior of a stronger model (Llama3.1-405B-Instruct) into a weaker model (Llama3.1-8B-Instruct). As a result, I don't find the performance improvements in this paper to be very surprising, and it's not clear to me that they teach us anything new. It would be nice to have clearer takeaways from the experimental work

questions:
1. As far as I can tell, the proposed task is less about actual travel planning (i.e., what is the quality of the proposed itinerary?) and more about preference elicitation and inference. In the real world analogue of this setting, it seems evident that many preferences can be elicited that aren't decision-relevant. How does your approach to evaluation account for this issue?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731135961032

thread_type:
official_review

<thread_object>
creator: reviewer_aaco

<content>
summary:
This paper introduces  full delegation, where individuals routinely entrust their routine decision-making to agents capable of making high-quality decisions in real-world scenarios tailored to personal needs. To achieve this, the paper has designed the APEC Agent Constitution, where APEC stands for Accuracy, Proactivity, Efficiency, and Credibility. The paper assesses both the process and the final outcome, and develops APEC-Travel to actively gather personalized travel preferences from users via multi-round dialogue. The experimental results on synthetic seed dialogues demonstrate impressive performance when compared to existing baselines.

strengths:
The paper is well-written and organized. It carefully selects 200 examples from the seed dialogues generated by the llama3.1-405b-instruct model, with human annotators evaluating them across five metrics: planning, prioritization, proactiveness, clarification, and helpfulness. The concrete measurement of each axis convincingly contributes to the construction of the final reward construction.

The experimental results show that APEC-Travel-DPO enhances agency scores and efficiency, leading to more concise dialogues with fewer interactive turns. 

The ablation study is comprehensive, covering various aspects and different hyper-parameter settings. Additionally, error analysis on 20 examples provides valuable insights into the factors contributing to low-quality dialogues.

weaknesses:
While guidelines are provided for human annotators on how to assess the dialogues, there is a lack of detailed information regarding the number of annotators involved, the process of ensuring consistency through cross-validation, and other measures taken to ensure the reliability of the final scores.

In the evaluation, although the test set comprises 1,000 examples with unique personas, there is a concern that all the seed dialogues (10k), reward construction examples (1k), and test examples (1k) are generated using the same llama3.1-405b-instruct model. This raises the possibility that the training and evaluation dialogues may still share similar distributions, except for persona variations.

questions:
Thanks for providing the basic training configurations for SFT and DPO in Tables 9 and 10. I noticed that increasing the number of iterations in DPO training could deteriorate performance. Could you please share the learning rates used for SFT and each stage of DPO? Additionally, do you have further insights beyond what is discussed in the "Iterative Training Paradigm" section?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper argues that agents should be evaluated beyond just task success rates and accuracy scores,
and proposes an "Agent Constitution" that aims to assess Accuracy, Proactivity, Efficiency and Credibility.
The reviewers agreed that the problem of aligning agents is well motivated and timely,
though there was significant disagreement about whether the paper's contributions are above the bar for publication at ICLR.
The reviewers appreciated the novelty of the proposal, and the demonstrated results on a travel planning task,
but were unconvinced by the transferability of the findings, as well as the necessity & sufficiency of the proposed constitution.

1. The paper states APEC as necessary dimensions for agent evaluation, as an axiom.
There needs to be more justification. There have been other principles for LLM-based chatbots (e.g. helpfulness and harmlessness, to name two).
How do the APEC principles relate to those studied for chatbots? What are the specific principles that are motivated by agentic behavior? Why can we not add those principles to the set of principles elucidated for chatbots?
Are APEC a sufficient set of dimensions for evaluating agents? Are there other principles that have not been captured, for instance, 
all side-effects caused by the agent are made apparent to the user (e.g. some form of scrutability).

2. Reviewers expected experiments on TravelPlanner datasets and more complex agent tasks, to build confidence that the proposed set of APEC principles
indeed capture desirable behavior across different agent benchmarks.