Sparse Rewards Can Self-Train Dialogue Agents

Authors: Barrett Martin Lattimer, Varun Prashant Gangal, Ryan McDonald, Yi Yang
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Self-training, LLM, Simulation, Benchmark, Tool-calling, Dataset, Task Oriented Dialogue, Dialogue, User Simulation, Beam Search, Algorithm, Preference Tuning, Supervised Finetuning, JOSH, ToolWOZ

Abstract:
Recent advancements in state-of-the-art (SOTA) Large Language Model (LLM) agents, especially in multi-turn dialogue tasks, have been primarily driven by supervised fine-tuning and high-quality human feedback. However, as base LLM models continue to improve, acquiring meaningful human feedback has become increasingly challenging and costly. In certain domains, base LLM agents may eventually exceed human capabilities, making traditional feedback-driven methods impractical. In this paper, we introduce a novel self-improvement paradigm that empowers LLM agents to autonomously enhance their performance without external human feedback. Our method, Juxtaposed Outcomes for Simulation Harvesting (JOSH), is a self-alignment algorithm that leverages a sparse reward simulation environment to extract ideal behaviors and further train the LLM on its own outputs. We present ToolWOZ, a sparse reward tool-calling simulation environment derived from MultiWOZ. We demonstrate that models trained with JOSH, both small and frontier, significantly improve tool-based interactions while preserving general model capabilities across diverse benchmarks. Our code and data are publicly available on GitHub.

Decision: reject
Reviewer scores: 5, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730258823154

thread_type:
official_review

<thread_object>
creator: reviewer_e3ly

<content>
summary:
This paper introduces JOSH, a self-training framework designed to enable agentic models to achieve self-alignment. The core component of JOSH is the data rollout pipeline, where an agent first interacts with a GPT-based simulator to generate multi-turn conversations that involve tool-calling responses. A critical aspect of this process is the use of beam search to create a tree-structured trajectory. From this trajectory tree, they extract SFT and preference data for subsequent fine-tuning. To evaluate JOSH, they have also curated a multi-turn tool-calling benchmark called ToolWOZ.

strengths:
The proposed method is evaluated on different model backbones, even gpt-4o

The curated benchmark is useful to the community.

The presentation is mostly clear and easy to follow.

weaknesses:
The primary weaknesses of this paper lie in its novelty and the experimental validation.

Novelty: The proposed framework is not particularly novel, as it builds upon concepts that have been extensively studied within the community. Techniques such as data rollouts, beam search, and supervised/preference fine-tuning have all been well-explored in prior works.

Experiments: This paper evaluates JOSH using only a single benchmark and does not provide comparisons with other robust baselines. There are numerous multi-turn tool-calling and agentic benchmarks available, such as WebLinx[1] and MINT[2]; conducting experiments on multiple benchmarks would significantly strengthen the validity of the results. Furthermore, there are several highly similar methods in this domain, such as [3,4], which should be considered as baselines to effectively demonstrate the true performance of JOSH.


[1] WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
[2] MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback
[3] Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents
[4] V-STaR: Training Verifiers for Self-Taught Reasoners

questions:
How much does it cost to finetune 4o?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679451640

thread_type:
official_review

<thread_object>
creator: reviewer_ykjk

<content>
summary:
They propose both a benchmark and a method for training multi-turn tool use dialogue agents. Their method uses beam search to find successful trajectories and uses failed paths in the beam search as negative examples. They finetune an LM on these successful/unsuccessful pairs with KTO. The benchmark is called ToolWOZ, which re-purposes the popular dialogue systems benchmark MultiWOZ to a more native LM tool use format. They evaluate their method on both ToolWOZ and another standard benchmark tau-bench. They find that their method substantially improves LLaMA 3-8B's success rate on both benchmarks. They also conduct evaluations of the robustness of each benchmark by analyzing the standard deviation of the results, finding ToolWOZ with the goal simulator to give the lowest standard deviation. Finally, they conduct some error analysis of their approach on ToolWOZ.

strengths:
* They propose both a novel method and a benchmark, but they also make sure to evaluate on an existing benchmark to enable more robust comparisons.
* They conduct good analysis to demonstrate the robustness and viability of their benchmark.
* Their method demonstrates good performance gains on the tasks they study.
* The paper is overall well written and easy to follow.

weaknesses:
* Their method feels a little ad-hoc. Yes, it makes sense to build off-policy preference pairs for training these models, but there are numerous ways this could be achieved and its unclear why the specific methodological decisions made in this paper are the correct ones.
* They compare to an SFT baseline, but not other RL-inspired approaches for finetuning agents, so it is unclear how well their approach compares against stronger baselines.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730779029322

thread_type:
official_review

<thread_object>
creator: reviewer_qkhy

<content>
summary:
This paper introduces a self-alignment approach called Juxtaposed Outcomes for Simulation Harvesting (JOSH), designed to improve dialogue agents in multi-turn, tool-calling tasks by leveraging sparse rewards. The authors propose ToolWOZ, a new simulation environment derived from MultiWOZ, for training agents to make correct API calls based on sparse reward feedback. The JOSH method aims to allow models, including smaller LLMs, to improve autonomously without relying on extensive human feedback, which is increasingly challenging to obtain as models advance.

strengths:
1. The JOSH approach is a new solution for self-training dialogue agents, effectively utilizing sparse rewards to build a self-improvement feedback loop without external human evaluation.
2. By adapting MultiWOZ into ToolWOZ with a sparse reward structure, the paper provides a valuable benchmark tailored for tool-using task-oriented dialogue systems, which can benefit further research.
3. Results indicate that JOSH significantly improves models across benchmarks, demonstrating its potential as a scalable solution for optimizing agent interactions in multi-turn dialogue settings.

weaknesses:
1. The concept of the "goal set" in sparse rewards is insufficiently defined, particularly how it influences the agent’s behavior and the implications of duplicating actions in a path.
2. The choice to branch at the turn level rather than the agent action level lacks a comprehensive rationale, leaving questions about its impact on computational efficiency and performance outcomes. In multiwoz dataset, the agent predicts dialogue act in each turn. The delexiclized response is then generated. The slots values are then filled in the delexiclized response to yield the final response. This process is clearly different from the one illustrated in Figure 2. 
3. While considerable effort is spent on detailing ToolWOZ, the sparse reward process and its precise mechanics within JOSH are not thoroughly elaborated, reducing clarity around its contribution to the results.
4. The baseline comparisons are primarily limited to supervised fine-tuning (SFT) and variants of the sparse reward approach itself. To better contextualize the efficacy of JOSH, comparisons with other RL-based methods, particularly those designed for dialogue or tool-calling tasks, would be beneficial.

questions:
As in weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1731118859140

thread_type:
official_review

<thread_object>
creator: reviewer_dqjx

<content>
summary:
The paper introduces JOSH (Juxtaposed Outcomes for Simulation Harvesting), a self-alignment framework for large language model (LLM) agents to enhance multi-turn dialogue capabilities without human feedback, addressing the impracticality of traditional feedback-driven methods. JOSH leverages sparse reward signals within simulated dialogues to allow the model to self-improve, specifically targeting multi-turn tool-calling skills in task-oriented dialogues. The authors also introduce ToolWOZ, a dataset and benchmark based on MultiWOZ 2.0, designed to evaluate tool-usage in dialogue settings. Experimental results demonstrate that a fine-tuned LLaMA-3B model exhibits a 74% increase in Success Rate, and gpt-4o also shows improvements following JOSH self-alignment. Additional experiments on other public benchmarks indicate that JOSH does not degrade the model’s general performance.

strengths:
* This paper presents a novel approach to self-alignment in dialogue agents using sparse rewards, reducing reliance on costly human feedback.
* ToolWOZ fills a gap in existing evaluation frameworks by focusing on tool usage in multi-turn dialogue settings, adapting MultiWOZ to emphasize real-world API interactions.
* JOSH demonstrates significant improvements in success rates and tool-call accuracy, particularly for smaller models, validating its effectiveness.

weaknesses:
* The paper does not assess how well the user simulator aligns with real human interactions.
* The evaluation of API calls lacks depth, as it does not separate analyses of API names and parameters.
* The design of the average reward function is not thoroughly examined, missing a discussion of alternative reward structures and their potential effects on agent behavior.
* The related work section does not cover relevant advancements in language agents for multi-turn dialogues.

questions:
* How does JOSH compare to other sparse reward-based alignment or self-improvement approaches?
* Could strategies like reflection, which are often beneficial for tree-search and multi-turn tasks, enhance JOSH’s effectiveness if integrated?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733193209692

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final Paper Draft Comments

comment:
Thank you to all of the reviewers for your insightful feedback and ongoing support. We've revised our paper to incorporate your suggestions and enhance its quality. Let us know if you have any further comments ahead of the comment deadline today.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work proposes an LLM self-training approach based on sparse reward simulation. The authors also plan to release their code and data. The reviewers appreciate the general idea of this work, the value of the ToolWOZ benchmark to the community, the performance improvements achieved, and the clarity in presentation. Concerns are also raised, however, primarily regarding experimental validation, the design of the simulator (validity, reward function and API call design) as well as concerns around novelty and relation to existing works. The authors provide extensive and detailed responses but the reviewers are not convinced.