Process-Supervised Reinforcement Learning for Interactive Multimodal Tool-Use Agents

## Abstract

Abstract Effective interactive tool use requires agents to master Tool Integrated Reasoning (TIR): a complex process involving multi-turn planning and long-context dialogue management. To train agents for this dynamic process, particularly in multimodal contexts, we introduce a sandbox environment for reinforcement learning (RL) that supports interleaved speech-text rollouts. Our core strategy, Turn-level Adjudicated Reinforcement Learning (TARL), addresses the challenge of credit assignment in long-horizon tasks by employing a Large Language Model (LLM) as a judge to provide turn-level evaluation. To enhance exploration, we integrate a mixed-task training curriculum with mathematical reasoning problems. This unified approach boosts the task pass rate on the text-based τ \tau -bench by over 6% compared to strong RL baselines. Crucially, we demonstrate our framework’s suitability for fine-tuning a multimodal foundation model for agentic tasks. By training a base multimodal LLM on interleaved speech-text rollouts, we equip it with tool-use abilities, paving the way for more natural, voice-driven interactive agents.

## 1 Introduction

Large Language Models (LLMs) [1, 2, 3, 4, 5] have demonstrated remarkable understanding and reasoning capabilities across diverse domains. As these models advance, enabling them to interact seamlessly with real-world tools and services has emerged as a promising direction. We aim to create agents that can understand and act upon not just text commands, but also spoken language, which requires a new paradigm for agent training. While interactions can span web interfaces, programming systems, and APIs, the fundamental challenge remains: the agent must interpret complex, often multi-turn user requests and execute appropriate actions, whether the input is typed or spoken.

To tackle this challenge, we focus on interactive tool-use agents. We build upon the experimental setup from $\tau$-bench ([figure˜2](https://arxiv.org/html/2509.14480v1#S2.F2)), where an agent assists a simulated user with complex tasks by strategically calling tools. We choose this interactive setting because it provides a direct path toward our primary goal: developing end-to-end voice agents that can act on spoken commands. This multi-turn conversational format realistically mirrors real-world applications and presents complex reasoning challenges even for state-of-the-art models.

To advance the state of these agents, we employ Reinforcement Learning (RL) as our primary training methodology. Unlike prior approaches [6] that rely on static, pre-collected trajectories, RL allows agents to learn from dynamic model rollouts in an online manner, which is crucial for handling the variability of real-world interactions. To support this RL-based training paradigm, we have developed a sandbox environment that facilitates agent interactions with users and tools through API calls using the Model Context Protocol [7, MCP]. A core feature of our infrastructure is its support for both text-based and audio-based user simulation, allowing us to train and evaluate both text-based and multimodal agents.

However, standard RL algorithms falter in this complex setting. We observed that as training progresses, models often become overconfident, reducing their capacity for exploration. To counteract this, we introduce a two-pronged strategy. First, we employ mixed-task training—incorporating medium-difficulty math problems—to encourage persistent exploration and regularize the learning process. Second, to solve the critical credit assignment challenge in our long multi-turn trajectories, we propose Turn-level Adjudicated Reinforcement Learning (TARL). This method uses an LLM-based judge to provide fine-grained, turn-level rewards that guide policy updates. On text-based tasks, the combination of these techniques boosted the pass rate by an additional 6% over our already strong RL baselines.

Having established our framework’s effectiveness in the text domain, we applied it to our main objective: training a multimodal agent with real-world utility. Leveraging our sandbox environment, we trained a base multimodal LLM on $\tau$-bench tasks with speech-based user simulation. Guided by our proposed mixed-task training and TARL strategies, our approach successfully equipped the model with robust interactive tool-use abilities, improving the pass rate by over 20% compared to the base model. This demonstrates a viable path for fine-tuning multimodal (and speech) foundation models for complex agentic tasks using process-supervised RL. In summary, our contributions are threefold:

- •
A generalizable, open-source sandbox designed for training interactive tool-use agents across both text and speech modalities.
- •
An enhanced RL training strategy (TARL) that improves performance by encouraging exploration and enabling fine-grained, turn-level credit assignment.
- •
The first demonstration of this framework to successfully train a multimodal voice agent through RL on interleaved speech-text interactions, showing great performance gains.

## 2 Preliminary

Figure: Figure 2: Our environment setup for interactive tool-use agents.
Refer to caption: x2.png

### 2.1 Sandbox Environment for Tool-Use Agents

Our sandbox environment, illustrated in [figure˜2](https://arxiv.org/html/2509.14480v1#S2.F2), comprises three components including (1) a backend application with a pre-configured database and API endpoints for MCP server communication; (2) a user simulator that leverages LLM capabilities to generate realistic user requests and responses; and (3) a rule-based verifier that evaluates interaction trajectories and provides binary rewards. Below, we detail each component’s implementation and functionality.

Backend Application. We implement a SQLite database to store the comprehensive dataset from $\tau$-bench, encompassing various data tables such as Products, Orders, and Users. Rather than relying on static JSON files for seed data storage, we construct a proper relational database schema with well-defined table structures and database operations. This design choice enables our backend application to be easily extended and adapted for other tasks. We expose the available tools as RESTful API endpoints through application routers and register them as MCP tools, providing seamless integration for agent interactions.

User Simulator. Our user simulator employs GPT-4 [1] to role-play as human users, generating contextually appropriate requests and responses based on the task instructions from $\tau$-bench. We adopt the ReACT [8] reasoning framework using a consistent prompt with $\tau$-bench, which compels the user model to engage in structured thinking processes before formulating responses to agent queries. For speech-based user simulation, we use SeedTTS [9] to convert the text responses to into natural speech.

Rule-based Verifier. We implement a rule-based verifier that systematically inspects successful write operations—specifically, tool calls that alter the database state, such as those involved in order modifications, exchanges, reservations, and cancellations. This verifier cross-references the arguments from the agent’s tool calls with ground-truth annotations and outputs a binary reward: 1 for a complete match and 0 otherwise.

Notably, $\tau$-bench includes an additional verification step that checks for expected outputs in the agent’s responses. However, we observe that this criterion is highly sensitive to variations in how user responses are phrased, so we exclude it from our reinforcement learning (RL) training and evaluation protocols. For the sake of consistency, though, we also report results incorporating this output check in LABEL:appendix::more_results.

### 2.2 RL Algorithms

We formulate the interactive tool-use agent training as a Markov Decision Process (MDP). Given an autoregressive language model as the policy backbone, the state at any point in the interaction is simply the token sequence observed so far. The interaction follows an alternating pattern: when the agent is responding (calling tools with arguments), it takes actions by sampling the next token from the policy distribution $p_{\theta}$ and append the token to existing trajectory. When the agent stops talking, the environment generates feedback (through simulated user agent or tool execution results) and append a sequence of tokens (denoting user response or tool execution result) to existing trajectory.

More formally, let $\bm{x}^{i}=(x^{i}_{1},x^{i}_{2},\ldots)$ denote the $i$-th agent token sequence and $\bm{e}^{i}=(e^{i}_{1},e^{i}_{2},\ldots)$ denote the $i$-th environment token sequence. When environment response is from tool execution or text-based user simulation, $\bm{e}^{i}$ is a sequence of text tokens. When we use speech-based user simulation, $\bm{e}^{i}$ is a sequence of speech tokens (or their placeholder tokens). The complete trajectory is an interleaved sequence: $\bm{\tau}=(\bm{x}^{1},\bm{e}^{1},\bm{x}^{2},\bm{e}^{2},\ldots,\bm{x}^{T},\bm{e}^{T})$. Here $T$ is the total number of interaction steps, reached when user agent replied special token ##STOP## or when the maximum number of interaction steps is reached. In our case, $T\in[1,30]$ as we set a maximum of 30 interaction steps. Our objective is to maximize the expected reward over complete trajectories:

$$ $J(\theta)=\mathbb{E}_{\bm{\tau}\sim p_{\theta}}[R(\bm{\tau})]$ (1) $$

where $R(\bm{\tau})$ is a trajectory-level reward function that evaluates the quality of the generated trajectory using the rule-based verifier described in [section˜2.1](https://arxiv.org/html/2509.14480v1#S2.SS1). To optimize this objective, we experiment with the following widely-used RL algorithms:

PPO (Proximal Policy Optimization):
We begin with PPO [10], which constrains policy updates to prevent large deviations from the current policy through a clipping mechanism. For brevity, we denote the conversation history up to token $x^{i}_{j}$ as $\bm{h}^{i}_{j}=[\bm{c};\bm{x}^{1},\bm{e}^{1},\ldots,\bm{x}^{i-1},\bm{e}^{i-1},x^{i}_{1},\ldots,x^{i}_{j-1}]$. PPO operates at the token level using policy gradient ratios. Given a current policy $\theta_{\text{old}}$ and a new policy $\theta$, the probability ratio for each token is:

$$ $r^{i}_{j}(\theta)=\frac{p_{\theta}(x^{i}_{j}|\bm{h}^{i}_{j})}{p_{\theta_{\text{old}}}(x^{i}_{j}|\bm{h}^{i}_{j})}$ (2) $$

The PPO objective function applies clipping to this ratio (we omit the KL divergence term here):

$$ $L^{\text{PPO}}(\theta)=\mathbb{E}_{\bm{\tau}\sim p_{\theta_{\text{old}}}}\left[\frac{1}{\sum_{i=1}^{T}|x^{i}|}\sum_{i=1}^{T}\sum_{j=1}^{|\bm{x}^{i}|}\min\left(r^{i}_{j}(\theta)A^{i}_{j},\text{clip}(r^{i}_{j}(\theta),1-\epsilon,1+\epsilon)A^{i}_{j}\right)\right]$ (3) $$

where $A^{i}_{j}$ is the advantage function and $\epsilon$ is the clipping parameter. PPO uses the Generalized Advantage Estimate (GAE):

$$ $A^{i}_{j}=\sum_{l=0}^{\infty}(\gamma\lambda)^{l}\delta^{i}_{j+l},\quad\text{where}\quad\delta^{i}_{j}=R^{i}_{j+1}+\gamma V(s^{i}_{j+1})-V(s^{i}_{j})$ (4) $$

This formula relies on several key terms. The calculation is driven by the Temporal Difference (TD) error ($\delta^{i}_{j}$), which measures the one-step prediction error of the value function. The TD error itself is found using the
immediate reward ($R_{j+1}^{i}$) and the value function’s estimate for the current and next states. This calculation is weighted by two parameters: the discount factor ($\gamma$), which determines how much future rewards are valued, and the GAE parameter ($\lambda$), which balances the trade-off between bias and variance in the final advantage estimate(^1^11 In practice, we set $\gamma=1,\lambda=1$ given our long-horizon trajectories). Since we use a trajectory-level verifiable reward, we simply have $R_{j+1}^{i}=R(\bm{\tau})$, i.e., the token-level reward is the same across all positions. Note that throughout our RL training, we compute loss only over agent sampled tokens $\bm{x}$ and mask the loss over all environment tokens $\bm{e}$ to avoid unstable updates.

GRPO (Group Relative Policy Optimization):
GRPO [11] also shares the same clipping mechanism but introduces a group-based relative policy optimization approach that normalizes rewards within each group to improve training stability. Given a group of $G$ trajectories $\{\bm{\tau}_{1},\bm{\tau}_{2},\ldots,\bm{\tau}_{G}\}$, GRPO computes the mean and standard deviation of rewards:

$$ $\mu_{R}=\frac{1}{G}\sum_{n=1}^{G}R(\bm{\tau}_{n}),\quad\sigma_{R}=\sqrt{\frac{1}{G}\sum_{n=1}^{G}(R(\bm{\tau}_{n})-\mu_{R})^{2}}$ (5) $$

The advantage for trajectory $n$ is then computed as $A_{n}=(R(\bm{\tau}_{n})-\mu_{R})/\sigma_{R}$. The GRPO objective function is:

$$ $L^{\text{GRPO}}(\theta)=\mathbb{E}_{\{\bm{\tau}_{n}\}_{n=1}^{G}\sim\mathbb{P}_{\theta_{\text{old}}}}\left[\frac{1}{\sum_{i=1}^{T_{n}}|x^{i}|}\sum_{i=1}^{T_{n}}\sum_{j=1}^{|\bm{x}^{i}|}\min\left(r^{i,n}_{j}(\theta)A_{n},\text{clip}(r^{i,n}_{j}(\theta),1-\epsilon,1+\epsilon)A_{n}\right)\right]$ (6) $$

where $r^{i,n}_{j}(\theta)=\frac{p_{\theta}(x^{i,n}_{j}|\bm{h}^{i,n}_{j})}{p_{\theta_{\text{old}}}(x^{i,n}_{j}|\bm{h}^{i,n}_{j})}$ is the probability ratio for token $j$ in turn $i$ of trajectory $n$. This normalization approach helps stabilize training by reducing reward scale variations across different batches and tasks.

RLOO (REINFORCE Leave-One-Out):
RLOO [12] is similar to GRPO but uses a different baseline computation. Given a group of $G$ trajectories $\{\bm{\tau}_{1},\bm{\tau}_{2},\ldots,\bm{\tau}_{G}\}$, the baseline for each trajectory $\bm{\tau}_{n}$ is computed using all other trajectories in the group:

$$ $b_{n}=\frac{1}{G-1}\sum_{j\neq n}R(\bm{\tau}_{j})$ (7) $$

Then the advantage function is computed as $A_{n}=R(\bm{\tau}_{n})-b_{n}$ for the $n$-th trajectory. This leave-one-out approach ensures that the baseline is unbiased while significantly reducing the variance of gradient estimates compared to standard REINFORCE.

### 2.3 Benchmark RL Algorithms

After constructing our sandbox environment, we first benchmark RL algorithms on $\tau$-bench to understand capabilities of vanilla RL algorithms on tool-use tasks. We utilize text-based user simulation with Qwen3-8B [5] as our base model with training configurations in LABEL:appendix::training_details. For our training data, we use GPT-4.1 to synthesize user instruction prompts and ground-truth tool-call annotations through publicly released trajectory data from APIGEN-MT [6] (^2^22 Publicly available at [https://huggingface.co/datasets/Salesforce/APIGen-MT-5k](https://huggingface.co/datasets/Salesforce/APIGen-MT-5k)) (Details in [section˜8](https://arxiv.org/html/2509.14480v1#S8)). Since there are very limited amount of trajectories and test cases for Airline, we only synthesize Retail domain’s training data. For evaluation, we assess our models on both Retail and Airline domains. Across all our experiments, we use the pass$\textasciicircum$k metric [13] in conjunction with our rule-based verifier. For a given task, pass$\textasciicircum$k equals 1 only when all k sampled conversation trajectories are verified as correct by the environment.

**Table 1: pass$\textasciicircum$k results of tool-use agents trained with different RL algorithms on $\tau$-bench (baselines models are replicated with our environment setup). $n$ denotes the number of rollouts sampled during RL training. The best RL-trained results are bolded. For the Retail domain, we also report #wait, Len: the average number of “wait” tokens (as an indicator of self-reflection) and response length per turn.**
| Agent Model | Retail | Airline |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| pass^1 | pass^2 | pass^3 | pass^4 | #Wait | Len | pass^1 | pass^2 | pass^3 | pass^4 |  |
| Baseline Models |  |  |  |  |  |  |  |  |  |  |
| GPT-4.1 | 60.9 | 55.7 | 51.3 | 47.8 | 0.3 | 54 | 48 | 34 | 26 | 24 |
| Llama-xLAM-2-8B [6] | 42.6 | 34.8 | 28.7 | 26.1 | 0.1 | 19 | 36 | 26 | 20 | 18 |
| Qwen3-8B [5] | 42.6 | 30.4 | 25.2 | 21.7 | 14.6 | 228 | 32 | 24 | 20 | 20 |
| Qwen3-8B + RL |  |  |  |  |  |  |  |  |  |  |
| GRPO (n=4) | 51.3 | 37.4 | 30.4 | 27.0 | 11.7 | 204 | 28 | 14 | 8 | 4 |
| RLOO (n=4) | 47.0 | 31.3 | 28.7 | 24.3 | 11.1 | 180 | 36 | 20 | 16 | 12 |
| PPO (n=4) | 48.7 | 36.5 | 31.3 | 26.1 | 8.4 | 162 | 32 | 22 | 14 | 12 |

In-Domain RL Training Shows Promise but Faces Limitations Our benchmark results in [table˜1](https://arxiv.org/html/2509.14480v1#S2.T1) demonstrate that all RL algorithms successfully improve Qwen3-8B’s performance on the Retail domain. GRPO achieves the largest improvement, closely followed by PPO (both using $n=4$ rollouts), indicating that RL training effectively enhances the model’s tool-using capabilities. The improvement is most pronounced in single-sample scenarios (pass^1), where GRPO delivers approximately 9% improvement over the baseline. However, as $k$ increases, the performance of RL-trained agents declines below 30%, revealing that while RL improves performance, sampling consistency remains limited for complex multi-turn tool-use tasks.

The generalization challenge becomes apparent when evaluating on out-of-domain Airline tasks. While RL-trained models perform comparably to the Qwen3-8B baseline when $k=1$, their performance deteriorates more rapidly as $k$ increases compared to the baseline model. This suggests that the exploration strategies learned through RL training do not transfer effectively to different domains. This limitation is expected given our training dataset of only 3,000 samples that only covers Retail domain. Achieving better generalization would require more diverse and challenging task formulations, as demonstrated by recent work like Kimi-K2 [14]. Since our focus lies in improving training mechanisms rather than data engineering, we concentrate on optimizing RL strategies for in-domain performance in the later sections.

The Confidence Paradox: When More Confidence Isn’t Better While RL training is known to enhance model confidence and sampling efficiency [11, 15, 16]—indeed reflected in our improved pass^1 results—this increased confidence comes with gradually reduced explorations. Analysis of our sampled trajectories reveals that post-training models exhibit reduced self-reflection and self-correction behaviors, as evidenced by the substantial decrease in “wait” tokens (Qwen3 tends to use phrases like ’wait, …’ to interrupt its thinking process and reflect on its actions) and shorter average response lengths. For example, we observe that the model over-confidently cancels orders without confirming with users, leading to avoidable errors.

Although these behavioral changes do not necessarily translate to lower overall performance, they significantly impact the exploration benefits of RL training once the model is confidently exploring sub-optimal strategies. Furthermore, the vanilla use of trajectory-level rewards could be problematic for multi-turn conversations—in our case, with contexts up to 32,768 tokens—as it creates sparse reward signals that lead to suboptimal credit assignment when the model performs multiple actions per trajectory. These challenges inspire us to design training strategies that encourage agent exploration with fine-grained turn-level feedback in the next section.

## 3 Method

To improve agent performance, we tackle the challenges of exploration and credit assignment in long conversations. We foster exploration through mixed-task training with Math problems, while simultaneously refining credit assignment with our proposed Turn-level Adjudicated Reinforcement Learning (TARL), which integrates turn-level evaluations into the reward signal for policy model training.

### 3.1 Mixed-Task Training

To encourage exploration during training, we propose incorporating medium-difficulty math problems into the training process. This strategy leverages the fact that base models like Qwen3-8B [5] have been pre-trained on mathematical and coding problems, giving them strong reasoning capabilities. When solving math problems, language models naturally engage in self-reflection and make multiple self-corrections, which elongates their chain-of-thought (CoT) [17] reasoning trajectories and promotes exploratory behavior. By mixing math problems with Retail domain tasks, we regularize the training process to prevent the model from overfitting to the retail domain while preserving its exploration abilities through self-reflection.

In practice, we evaluated several math datasets including GSM8K [18], DeepScaleR [19], and DAPO-MATH-17K [20], ultimately selecting medium-difficulty problems from DeepScaleR. We chose this dataset because medium-level problems provide sufficient challenge to force the model to reflect on its reasoning process and generate longer CoT trajectories, while remaining manageable difficulty for an 8B parameter model.

### 3.2 Turn-level Adjudicated Reinforcement Learning

To provide fine-grained assessment of multi-turn trajectories, we employ an LLM-based judge that assigns turn-level rewards, as visualized in [figure˜3](https://arxiv.org/html/2509.14480v1#S3.F3). Each dialogue turn consists of two components: (1) the agent’s reasoning process and tool execution following ReACT format, and (2) environment feedback (treating user interactions as a special type of tool call, with user responses serving as environment feedback). The judge evaluates each dialogue turn after receiving the full conversation history with ground-truth annotations. The detailed prompt for the judge is provided in LABEL:appendix::llm_judge.

Our scoring mechanism assigns each turn one of three rewards: $-1$, 0, or $1$, with the important constraint that at most one turn can receive $-1$ per trajectory. A reward of $-1$ indicates a major deviation from expected behavior, typically occurring when the agent provides incorrect information after faulty reasoning (e.g., selecting the wrong item during an exchange request) or executes tool calls with erroneous arguments that cause irreversible database changes (e.g., canceling orders that should not be canceled). A reward of 0 indicates minor issues that are later corrected or a by-product of major deviation. A turn receives $1$ for correct execution without issues.

After obtaining turn-level rewards $r_{i}\in\{-1,0,1\},i\in[1,T]$ and the trajectory-level terminal reward $R(\bm{\tau})\in\{0,1\}$ from our rule-based verifier, we re-scale and combine them to better separate trajectories into distinct quality categories. For the binary terminal reward $R(\bm{\tau})$, we scale it by $10\times$ to emphasize task completion. For turn-level rewards, major deviations ($-1$) are scaled by $5\times$ to heavily penalize critical errors, while other turns are scaled by $\frac{1}{T}$ so that the maximum contribution from turn-level rewards is capped at 5 points regardless of trajectory length. This scaling ensures that longer trajectories are not unfairly advantaged and emphasizes penalty from the major deviation. Our reward design yields four distinct trajectory categories:

- 1.
Perfect trajectory ($15$ points): $10$ points for terminal success plus $5$ points from turn-level rewards.
- 2.
Good trajectory ($10-15$ points): $10$ points for terminal success plus $0-5$ points from turn-level rewards, indicating some turns have minor issues.
- 3.
Good attempt trajectory ($0-5$ points): 0 points for terminal failure but positive turn-level rewards, occurring when the judge finds no major errors despite rule-based verification failure (rare cases, often due to unclear or hallucinated user responses)
- 4.
Failed trajectory ($-5$ to 0 points): 0 points for terminal failure plus $-5$ points for one major error, with some positive reward from other turns.

Figure: Figure 3: To provide turn-level reward, the judge assesses each turn based on the full conversation history with ground-truth annotations. In this example, model makes mistake by picking the wrong bluetooth speaker variant, which deviate the conversation from the expected trajectory.
Refer to caption: x3.png

We integrated our turn-level reward system into two reinforcement learning algorithms, GRPO and PPO, which both demonstrated strong performance on our benchmarks. For GRPO, which operates at the trajectory level, the integration is straightforward. We simply sum the individual turn-level rewards with the final terminal reward to calculate a total score for the entire trajectory. This score is then used to compute the advantage based on the mean and variance across rollouts (see [Equation 5](https://arxiv.org/html/2509.14480v1#S2.E5)). Integrating with PPO is more nuanced, as it calculates advantages at the token level. We tested two different strategies:

- •
Per-Turn Assignment: Applying each turn’s reward specifically to the final token of that turn to provide more granular feedback, which will be propogated backwards by GAE (see [Equation 4](https://arxiv.org/html/2509.14480v1#S2.E4)).
- •
Trajectory-Level Assignment: Calculating a single, normalized reward for the entire trajectory and applying it uniformly across all tokens.

Our later experiments showed that the trajectory-level assignment performs much better. The per-turn approach led to unstable training, while applying a uniform reward across all tokens yielded much more stable and effective results. Beyond our core reward design, we also attempted several other strategies, including encouraging exploration with high-entropy token training [21] and utilizing turn-level verifiers to interrupt the reasoning process and force self-reflection. Though these strategies did not yield improvements, we discuss them in our analysis ([section˜5](https://arxiv.org/html/2509.14480v1#S5)) to provide insights for future research.

## 4 Experiments

### 4.1 Text-based Agent Setup

Training Data. We train our text-based agents on approximately 3,000 synthetic tasks derived from APIGEN-MT [6] trajectories. Each task provides: (1) a user instruction to guide the simulated user, and (2) the ground-truth tool calls the agent is expected to execute. An example is provided in [section˜8](https://arxiv.org/html/2509.14480v1#S8) (LABEL:tab::dataset_example). To ensure comprehensive coverage, our sandbox environment is also pre-populated with all seed data from $\tau$-bench. For our mixed-task training strategy, we incorporate math problems from the DeepScaleR dataset [19], filtering for problems with integer answers and alternating between Retail and math tasks during training. We will open-source all curated task instructions and ground-truth tool calls.

Evaluation. Performance is evaluated on the $\tau$-bench Retail domain test set using the pass$\textasciicircum$k metric [13].

Model. We use Qwen3-8B [5] as the base model for our experiments. When using our proposed Turn-level Adjudicated Reinforcement Learning (TARL), we employ GPT-4.1 as the LLM judge to score each turn, following the mechanism described in [section˜3.2](https://arxiv.org/html/2509.14480v1#S3.SS2). For full training hyper-parameters, please refer to LABEL:appendix::training_details.

### 4.2 Text-based Agent Training Results

We present our main results in [table˜2](https://arxiv.org/html/2509.14480v1#S4.T2), where we see that our proposed Turn-level Adjudicated Reinforcement Learning (TARL) achieves better performance, which is further improved when we apply mixed-task training with Math problems. Ultimately, our best strategy (Math + TARL) demonstrates consistent improvements over standard reinforcement learning (RL) baselines, such as vanilla post-training with GRPO or PPO(^3^33 Here TARL utilizes the trajectory-level assignment for PPO. Ablation on reward granularity is conducted in [section 5](https://arxiv.org/html/2509.14480v1#S5)).

On the pass$\textasciicircum$1 metric, our optimal method (Math+TARL) achieves a 6% relative improvement over GRPO and a 15% improvement over the base model, reaching a final score of 57.4%. This result is competitive with capable closed-source models like GPT-4.1 (see [table˜1](https://arxiv.org/html/2509.14480v1#S2.T1)). Moreover, we observe consistent performance gains across different values of $k$, indicating enhanced reliability in completing challenging tasks.

Qualitatively, our method also produces models that engage in more frequent self-correction and generate longer responses compared to vanilla RL algorithms, as shown by the average number of “wait” tokens (#Wait) and response length (Len) metrics in [table˜2](https://arxiv.org/html/2509.14480v1#S4.T2). In our analysis ([section˜5](https://arxiv.org/html/2509.14480v1#S5)), we further compare training statistics (response length, KL divergence, training reward) of these strategies along with some failed approaches to provide more insights.

**Table 2: Performance comparison of different training strategies on the $\tau$-bench Retail domain. We report average wait time (#Wait), average response length (Len), and pass^k metrics. Our proposed strategies (highlighted rows) achieve the best performance, consistently outperforming already strong RL baselines.**
| Agent Model | Response Metrics | Performance Metrics |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | #Wait | Len | pass^1 | pass^2 | pass^3 | pass^4 |
| Baseline Model |  |  |  |  |  |  |
| Qwen3-8B | 14.6 | 228 | 42.6 | 30.4 | 25.2 | 21.7 |
| Qwen3-8B + RL (GRPO Variants) |  |  |  |  |  |  |
| GRPO | 11.7 | 204 | 51.3 | 37.4 | 30.4 | 27.0 |
| +TARL | 14.0 | 210 | 53.9 (+2.6) | 40.9 (+3.5) | 33.9 (+3.5) | 30.4 (+3.4) |
| +MATH + TARL | 15.8 | 236 | 57.4 (+6.1) | 42.6 (+5.2) | 36.5 (+6.1) | 33.9 (+6.9) |
| Qwen3-8B + RL (PPO Variants) |  |  |  |  |  |  |
| PPO | 8.4 | 162 | 48.7 | 36.5 | 31.3 | 26.1 |
| +MATH + TARL | 11.5 | 204 | 53.0 (+4.3) | 40.0 (+3.5) | 35.7 (+4.4) | 31.3 (+5.2) |

### 4.3 Multimodal Agent Setup

Environment and Simulation To extend our framework to voice-driven interaction, we simulate realistic user speech by first generating textual user prompts and then converting them to audio using SeedTTS [9], a high-quality text-to-speech model. This allows us to train agents on interleaved speech-text rollouts.

Evaluation. For evaluation, we assess the model in both text and speech modes. For the text mode, all settings are the same as text agents. For speech-mode evaluation, we exclude the authentication step from the Retail task in $\tau$-bench, as this step requires the agent to obtain user IDs in "name_number" format, which proves error-prone when processed through our TTS pipeline. Instead, we directly provide the agent with the user profile and continue the conversation.

Model Selection and Baseline Performance Our first step was to select a suitable base model capable of processing both speech and text. We evaluated several state-of-the-art foundational models, including Qwen2.5-Omni [22], Audio-Flamingo3 [23], and Audio-Reasoner [24]. We found that none of these models demonstrated satisfactory tool-use capabilities out-of-the-box. While Audio-Flamingo3 and Audio-Reasoner struggled significantly, often hallucinating after one or two turns, Qwen2.5-Omni-7B achieved the best—though still poor—initial performance with a pass^1 rate of 7.8% (see [table˜3](https://arxiv.org/html/2509.14480v1#S4.T3)). This highlights that multi-turn, interactive tool-use remains an under-explored capability for most speech-enabled foundation models.

Curriculum Learning for Warming Up Given the models’ limited initial abilities, we adopted a curriculum learning strategy to warm-up the multimodal agent’s tool-use abilities. Instead of supervised fine-tuning, we applied GRPO for 30 steps using a simplified set of training tasks. These tasks feature more detailed and specific user instructions to create a easier learning environment for skill acquisition (see [section˜8](https://arxiv.org/html/2509.14480v1#S8)). Qwen2.5-Omni showed rapid improvement on this simplified curriculum, demonstrating its ability to generate coherent trajectories and learn from interaction.

After the curriculum learning phase, we train the model on our normal training dataset and evaluate its performance across both text and speech modalities. During training, we employ a mixed-modality training strategy where the dataloader alternates between three types of data batches: (1) math problems, (2) text-only Retail task, and (3) Retail task with user response in speech. The first two batch types follow the same configuration used when post-training text agents. For the third data type, the model explores with interleaved speech-text rollouts where the speech contents are generated by the simulated user agent.

**Table 3: Performance comparison across training and evaluation modalities on $\tau$-bench. Models are trained with speech-text (S-T) or text-only (T-only) rollouts and evaluated with text or speech-based user agent. Our proposed methods (highlighted rows) achieve the best performance.**
| Training Configuration | Performance Metrics |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Eval Mode | Agent | Train Mode | pass^1 | pass^2 | pass^3 | pass^4 |
| Baseline Models |  |  |  |  |  |  |
| Text | Qwen2.5-Omni-7B | — | 7.8 | 7.8 | 7.8 | 7.8 |
| Speech | Qwen2.5-Omni-7B | — | 14.8 | 8.7 | 5.2 | 5.2 |
| Qwen2.5-Omni-7B + RL |  |  |  |  |  |  |
| Text | GRPO + Math | S & T | 31.3 | 20.9 | 12.2 | 12.2 |
|  | GRPO + Math + TARL | S & T | 36.5 | 25.2 | 21.7 | 16.5 |
| Speech | GRPO + Math | S & T | 34.8 | 25.2 | 21.7 | 16.5 |
|  | GRPO + Math + TARL | S & T | 37.4 | 26.1 | 22.6 | 20.9 |
|  | GRPO + Math + TARL | T-only | 32.2 | 18.3 | 14.8 | 11.3 |

### 4.4 Multimodal Training Results

The results presented in [table˜3](https://arxiv.org/html/2509.14480v1#S4.T3) demonstrate the effectiveness of our approach. GRPO + Math + TARL consistently delivers superior performance across both text and speech evaluation settings, validating the robustness of our method across modalities. Compared to the baseline model, our post-trained variant achieves more than 20% improvement in pass^1 performance, highlighting the benefits of our training strategies. Though the performance from the post-trained multimodal agent is still lacking behind the text agents, we anticipate greater performance gains as multimodal LLMs continue to improve.

Note that we also conduct an ablation study where the model is trained exclusively on text modality (using identical settings as our previous Qwen3-8B post-training experiments) and evaluated with speech-based user agent. The results, shown in the final row of [table˜3](https://arxiv.org/html/2509.14480v1#S4.T3), reveal degraded performance compared to the model trained with interleaved speech-text (S-T) rollouts. This degradation indicates that the model could lose its pre-trained speech understanding capabilities when fine-tuned solely on textual data, underscoring the critical importance of mixed-modality training for developing effective voice agents.

## 5 Analysis

To investigate methods for improving agent performance, we utilized our text-based agent, selected for its better reasoning capabilities and lower training cost. Our analysis, which focused on incentivizing exploration and credit assignment, revealed that simple and effective techniques are more effective than complex ones.

### 5.1 Reward Granularity for PPO-based Training

Given that PPO supports token-level rewards, we investigate how different reward granularities affect training performance. After obtaining turn-level evaluation from our judge, we experiment with two distribution strategies: (1) TARL (turn-level): assigning per-turn rewards at the final token position of each turn to provide fine-grained feedback, and (2) TARL (trajectory-level): computing a single trajectory-level reward using GRPO’s group normalization formula and applying it uniformly across all token positions.

Figure: Figure 4: Training time average reward and response length comparison of PPO-based training strategies. Aggregated trajectory-level reward with turn-level evaluation (TARL traj-level) obtains the best performance.
Refer to caption: x4.png

As illustrated in [figure˜4](https://arxiv.org/html/2509.14480v1#S5.F4), the trajectory-level approach promotes more effective exploration and exhibits stable reward growth during training, ultimately achieving a 4.3% improvement in pass^1 performance compared to vanilla PPO training (see [table˜2](https://arxiv.org/html/2509.14480v1#S4.T2)). In contrast, assigning rewards at turn-level granularity leads to degraded performance, with training rewards falling below even the vanilla PPO baseline. We hypothesize that assigning turn-level rewards at different positions complicate the credit assignment process and overly rely on the judge’s accuracy. For example, the turns after major deviation will receive little penalty and it might destablize the training if the judge’s turn is not accurate. On contrary, trajectory-level reward is much more robust as they are broadly disected into four categories outlined in [section˜3.2](https://arxiv.org/html/2509.14480v1#S3.SS2).

### 5.2 Strategies for Incentivizing Exploration

Data Distribution Modification: Mixed-Task Training  We first examine the effectiveness of mixed-task training with mathematical problems. As shown in [figure˜5](https://arxiv.org/html/2509.14480v1#S5.F5), GRPO+Math demonstrates increased exploration activity during training, evidenced by longer average response lengths compared to the baseline. However, despite this enhanced exploration, test set performance remains comparable to the GRPO baseline, suggesting that exploration alone is insufficient for improved generalization. The combination of exploration strategies with better credit assignment proves crucial. GRPO+Math+TARL, which incorporates both mixed-task training and turn-level rewards, exhibits the highest exploration levels (reflected in the longest average response lengths) and achieves substantially better performance on test set tasks ([table˜2](https://arxiv.org/html/2509.14480v1#S4.T2)). Notably, all methods—GRPO, GRPO+Math, and GRPO+Math+TARL—converge to similar high reward levels during training ([figure˜3](https://arxiv.org/html/2509.14480v1#S3.F3)), indicating that the benefits of enhanced exploration and credit assignment primarily manifest in generalization to unseen Retail tasks rather than training performance improvements.

Figure: Figure 5: Training time average reward and response length comparison of different training strategies. Mixed-task training with fine-grained turn-level evaluation (GRPO+Math+TARL) achieves the best performance.
Refer to caption: x5.png

Loss Function Adjustment: Entropy-based Modification  We are also curious if loss function adjustment with entropy-based modification could help incentivize exploration. We follow the recent study [21] to restrict policy gradient updates to the top 20% highest-entropy tokens. While this approach shows improved exploration compared to the baseline (see GRPO+Entropy+TARL in [figure˜5](https://arxiv.org/html/2509.14480v1#S5.F5)), it fails to enhance test-time performance and actually achieves lower training rewards than other strategies. We hypothesize that though entropy-based modification helps the model to explore, limiting updates to high-entropy positions could cause training instability, particularly problematic for our long-horizon sequential decision-making tasks.

Rollout Modification: Turn-level Intervention  We also explored whether real-time intervention could improve the agent’s exploration and self-reflection. Inspired by self-refinement techniques [25, 26], we deployed an LLM-based verifier to continuously monitor the agent’s action, as visualized in [figure˜6](https://arxiv.org/html/2509.14480v1#S5.F6).

Figure: Figure 6: Rollout intervention with an LLM-based judge. When major deviations from expected ground-truth tool calls are detected, the judge will force the agent to think and act again.
Refer to caption: x6.png

By comparing the agent’s actions to ground-truth tool calls, the verifier can identify suboptimal reasoning as it happens. When a mistake is detected, it triggers a self-correction mechanism by interrupting the generation and adding a corrective prompt to the reasoning trace: “Wait, my previous reasoning might be wrong, let me try again.” This prompts the model to find a better approach, with a limit of two interventions per reasoning step to avoid infinite loops. This real-time guidance is a more dynamic version of our turn-level reward system, which only provides feedback after a task is complete. However, this strategy ultimately backfired, destabilizing training without improving performance. As shown in [figure˜7](https://arxiv.org/html/2509.14480v1#S5.F7), the rapid decrease in entropy loss, paired with a significant rise in KL divergence, suggests the model began to overfit the unusual data patterns created by the interruptions. We attribute this failure to the disruption of the model’s natural thought process, which led to confusion and worse results than even the standard baseline.

Figure: Figure 7: Training time average reward, entropy loss, and KL loss comparison between GRPO and GRPO+Intervention.
Refer to caption: x7.png

Key Takeaways:
Our mixed-task training strategy, when combined with a trajectory-level assessment that integrates both turn-level and terminal rewards, promotes more effective exploration and yields higher task completion rates. In contrast, more sophisticated interventions like complex reward shaping and elaborate training loss designs tend to destabilize the training process and ultimately degrade performance—a finding that echoes the "bitter lesson" [27].

## 6 Related Work

### 6.1 Tool-Use Agent Benchmarks

Numerous evaluation benchmarks have been developed for tool-use tasks, including $\tau$-bench [13, 28], BFCL [29], AppWorld [30], ToolSandbox [31], UserBench [32], and Ace-Bench [33]. In our work, we adopt $\tau$-bench for training and evaluation as it supports realistic user-agent interactions. The user simulation aspect is also suitable for testing an end-to-end voice agent. However, $\tau$-bench has limitations, including its narrow scope of tasks (supporting only 2 domains) and limited control over user behavior. More recent benchmarks like UserBench have begun addressing these issues through preference-driven interactions, and we expect continued work in this direction to provide more realistic and controllable sandboxes for tool-use tasks.

### 6.2 Training Tool-Use Agents

Reinforcement learning (RL) algorithms have been developed and tested on a wide spectrum of problems. Foundational work demonstrated success in classic control tasks and games, such atari games [34], and AlphaGo [35]. More recently, RL has become a cornerstone for refining large language models (LLMs) beyond standard pre-training. Techniques like Reinforcement Learning from Human Feedback (RLHF) were critical in aligning models to follow user instructions and enhance safety [36]. This paradigm has been extended to improve complex reasoning abilities, such as solving mathematical problems by rewarding correct final outcomes [11] or verifying intermediate reasoning steps with process reward modeling [37]. RL has also been applied to agentic tasks, such as WebShop [38, 39, 40], AppWorld [41], etc., with a simulated environment.

Addressing the credit assignment challenge in multi-turn interactions is difficult when using only final outcome-based rewards, despite their scaling potential [11, 42]. Recent studies have shown that turn-level feedback offers a more effective solution for tool-use agents [43, 44, 45]. Building on insights from Process Reward Modeling (PRM) [37, 46, 42, 47], we implement a turn-level reward system. Unlike previous approaches that rely on structured, rule-based evaluators [44, 43], our method employs an LLM as a judge to provide more nuanced feedback (such as distinguishing between small and recoverable error versus major deviation) on an agent’s performance at each turn.

## 7 Conclusion

We develop an interactive tool-use agent that communicates with simulated users and tool sandboxes to complete complex tasks. Through our carefully crafted environment, we enable the agent to perform online exploration and train it using reinforcement learning algorithms. We further enhance the learning process by incorporating mixed-task training to sustain exploration and employing turn-level evaluation to improve credit assignment in long-horizon tasks. Furthermore, we extend our framework to train multimodal voice agents, incorporating additional strategies such as curriculum learning and mixed-modality training to enhance agent performance across different modalities.

## References

- [1]
OpenAI.
Gpt-4 technical report, 2024.
- [2]
Meta AI.
The llama 3 herd of models, 2024.
- [3]
Anthropic.
The claude 3 model family: Opus, sonnet, haiku.
- [4]
Gemini Team.
Gemini: A family of highly capable multimodal models, 2025.
- [5]
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu.
Qwen3 technical report, 2025.
- [6]
Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, Juan Carlos Niebles, Shelby Heinecke, Weiran Yao, Huan Wang, Silvio Savarese, and Caiming Xiong.
Apigen-mt: Agentic pipeline for multi-turn data generation via simulated agent-human interplay, 2025.
- [7]
Model context protocol.
[https://modelcontextprotocol.io/docs/getting-started/intro](https://modelcontextprotocol.io/docs/getting-started/intro), 2025.
- [8]
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models, 2023.
- [9]
Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, Mingqing Gong, Peisong Huang, Qingqing Huang, Zhiying Huang, Yuanyuan Huo, Dongya Jia, Chumin Li, Feiya Li, Hui Li, Jiaxin Li, Xiaoyang Li, Xingxing Li, Lin Liu, Shouda Liu, Sichao Liu, Xudong Liu, Yuchen Liu, Zhengxi Liu, Lu Lu, Junjie Pan, Xin Wang, Yuping Wang, Yuxuan Wang, Zhen Wei, Jian Wu, Chao Yao, Yifeng Yang, Yuanhao Yi, Junteng Zhang, Qidi Zhang, Shuo Zhang, Wenjie Zhang, Yang Zhang, Zilin Zhao, Dejian Zhong, and Xiaobin Zhuang.
Seed-tts: A family of high-quality versatile speech generation models, 2024.
- [10]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms, 2017.
- [11]
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
- [12]
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker.
Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms, 2024.
- [13]
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.
$\tau$-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024.
- [14]
Kimi Team.
Kimi k2: Open agentic intelligence, 2025.
- [15]
Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, and Jacob Andreas.
Beyond binary rewards: Training lms to reason about their uncertainty.
arXiv preprint arXiv:2507.16806, 2025.
- [16]
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang.
Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025.
- [17]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.
Chain-of-thought prompting elicits reasoning in large language models, 2023.
- [18]
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.
Training verifiers to solve math word problems, 2021.
- [19]
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica.
Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl.
[https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Preview-with-a-1-5B-Model-by-Scaling-RL-19681902c1468005bed8ca303013a4e2](https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Preview-with-a-1-5B-Model-by-Scaling-RL-19681902c1468005bed8ca303013a4e2), 2025.
Notion Blog.
- [20]
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang.
Dapo: An open-source llm reinforcement learning system at scale, 2025.
- [21]
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin.
Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning, 2025.
- [22]
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin.
Qwen2.5-omni technical report, 2025.
- [23]
Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and Bryan Catanzaro.
Audio flamingo 3: Advancing audio intelligence with fully open large audio language models, 2025.
- [24]
Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, and Chunyan Miao.
Audio-reasoner: Improving reasoning capability in large audio language models, 2025.
- [25]
Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao.
Large language models are better reasoners with self-verification, 2023.
- [26]
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark.
Self-refine: Iterative refinement with self-feedback, 2023.
- [27]
Richard S. Sutton.
The Bitter Lesson.
[https://www.cs.utexas.edu/˜eunsol/courses/data/bitter_lesson.pdf](https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf), 3 2019.
- [28]
Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan.
$\tau^{2}$-bench: Evaluating conversational agents in a dual-control environment, 2025.
- [29]
Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez.
The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models.
In Forty-second International Conference on Machine Learning, 2025.
- [30]
Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian.
Appworld: A controllable world of apps and people for benchmarking interactive coding agents, 2024.
- [31]
Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang.
Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities, 2025.
- [32]
Cheng Qian, Zuxin Liu, Akshara Prabhakar, Zhiwei Liu, Jianguo Zhang, Haolin Chen, Heng Ji, Weiran Yao, Shelby Heinecke, Silvio Savarese, Caiming Xiong, and Huan Wang.
Userbench: An interactive gym environment for user-centric agents, 2025.
- [33]
Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Yuefeng Huang, Wulong Liu, Xinzhi Wang, Defu Lian, Baoqun Yin, Yasheng Wang, and Wu Liu.
Acebench: Who wins the match point in tool usage?, 2025.
- [34]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.
Human-level control through deep reinforcement learning.
Nature, 518(7540):529–533, 2015.
- [35]
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.
Mastering the game of go with deep neural networks and tree search.
Nature, 529(7587):484–489, 2016.
- [36]
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
Training language models to follow instructions with human feedback.
Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
- [37]
Hunter Lightman, Vineet Kosaraju, Yura Reask, Ashish Soni, Collin Baker, Reecha Tiwari, Tony Jiang, Michael Laskin, Greg Brockman, Ilya Sutskever, et al.
Let’s verify step by step.
arXiv preprint arXiv:2305.20050, 2023.
- [38]
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan.
Webshop: Towards scalable real-world web interaction with grounded language agents.
In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 20744–20757. Curran Associates, Inc., 2022.
- [39]
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar.
Archer: Training language model agents via hierarchical multi-turn rl, 2024.
- [40]
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov.
Agent q: Advanced reasoning and learning for autonomous ai agents, 2024.
- [41]
Kevin Chen, Marco Cusumano-Towner, Brody Huval, Aleksei Petrenko, Jackson Hamburger, Vladlen Koltun, and Philipp Krähenbühl.
Reinforcement learning for long-horizon interactive llm agents, 2025.
- [42]
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin.
The lessons of developing process reward models in mathematical reasoning, 2025.
- [43]
Weikang Zhao, Xili Wang, Chengdi Ma, Lingbin Kong, Zhaohua Yang, Mingxiang Tuo, Xiaowei Shi, Yitao Zhai, and Xunliang Cai.
Mua-rl: Multi-turn user-interacting agent reinforcement learning for agentic tool use, 2025.
- [44]
Siliang Zeng, Quan Wei, William Brown, Oana Frunza, Yuriy Nevmyvaka, and Mingyi Hong.
Reinforcing multi-turn reasoning in llm agents via turn-level credit assignment, 2025.
- [45]
Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li.
Sweet-rl: Training multi-turn llm agents on collaborative reasoning tasks, 2025.
- [46]
Qianli Ma, Haotian Zhou, Tingkai Liu, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang.
Let’s reward step by step: Step-level reward model as the navigators for reasoning.
arXiv preprint arXiv:2310.10080, 2023.
- [47]
Sanjiban Choudhury.
Process reward models for llm agents: Practical framework and directions, 2025.

## 8 Training Data

| User Specification | Ground-Truth Tool-Calls |
| --- | --- |
| Your email is noah.brown7922@example.com. You are a customer who recently received an order and want to exchange two items: the green small polyester laptop-compartment backpack for a navy large polyester laptop-compartment backpack, and the black wired laser gaming mouse for a black wireless optical gaming mouse … Describe the items you want to exchange and the new options you want, and confirm the use of your original payment method for any price difference only after the agent identifies it. Respond to the AI agent to complete your exchange request. |  |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. What is the performance impact when the LLM is allowed to assign a score of -1 to more than one turn in a conversation? This seems like a limiting design choice, but I am curious if it is important for performance. 

2. How important is the warm-up curriculum? If it is important, it would be an interesting result and probably can be included in the proposed approach. 

3. The final trajectory-level reward involves 10x scaling of the turn-level scores. What is the sensitivity of performance to this particular scaling?
soundness: 3
strengths: 1. The authors are (to my knowledge), the first to evaluate a multimodal LLM agent trained on speech. 

2. The mixed-training objective, while simple, proves to be an effective method at overcoming the LLM agent becoming too overconfident during RL training. 

3. The authors demonstrate that turn-level rewards perform more effectively than trajectory-level ones, which will prove useful for the development of general LLM agents in multiturn environments.
summary: The paper addresses the challenge of training LLM agents for tool-use in multi-turn environments. The authors propose the following modifications: (1) training on a mixture of task-specific and general reasoning problems (specifically in the math domain) to encourage exploration, and (2) using LLM-as-a-judge for turn-level rewards. The authors evaluate their approach on base textual \tau-bench, and then on a multi-modal version using speech.
weaknesses: 1. The method relies heavily on LLM-as-a-judge, specifically the identify the most important mistake made in a trajectory. This seems like a limiting heuristic, and it is unclear if LLMs are capable of analyzing multiturn trajectories in this manner, when they contain multiple, cascading errors. My biggest concern with the method is that the current way turn-level rewards are labeled does not seem generalizable to other multiturn domains.  

2. The evidence that mixed-task training improves exploration seems inconclusive. It would be more convincing if the number of training samples were kept consistent between mixed-task and single-task training. Namely, if some samples of training on \tau-bench were instead co-opted with training on Math, would the benefit due to improved exploration still be visible?

3. Overall, the proposed method itself seems very specific, with the heuristic reward function as well as the training on math. The results would be stronger with an ablation on the reward function, as well as a more comprehensive analysis of how important mixed-training is, e.g. by considering more tasks other than math.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: - Can we replace the LLM judges with verifiable sub-goals where possible? 
- Do TARL gains hold with real world noisy speech? 
- What's the impact on latency in the multimodal pipeline with TTS models in the loop? 
- Testing on policy adherence will clarify if the RL based training system retain various refusal or other policy driven behaviors under TARL based training - did the authors test this aspect?
soundness: 3
strengths: -  this paper interleaves speech and text modalities for RL pipeline for improving tool use (necessary for a successful outcome), showing improvements on an agentic benchmark dataset. Authors claim it’s the first demonstrated training of a multimodal voice agent via RL on interleaved speech–text tool use. Although most ingredients of the system exist, but the combination of several of these ingredients is the main strength and contribution of this work. 
- other aspects such as RL for tool use within a sandbox environment and turn level supervision with LLM judges is incremental strength of this research; there are multiple systems, models and frameworks that exist today (eg. Kimi-K2 has very impressive tool use results on multiple tool-use benchmarks); this work focuses on enabling smaller models (Qwen 7B variants) which is a welcome direction for many researchers in the field who can easily build up research on top of these model sizes and systems.
summary: This paper focuses on training interactive tool use agents by creating (i) a sandbox environment with text based and audio based user simulators, (ii) database of tasks from retail and airline domain (based on τ-bench) and (iii) utilizing turn based and outcome based rewards. The authors utilize a mixed training strategy by interleaving medium difficulty math problems with the domain tasks to encourage the models to reason, explore, self-critique and correct. Authors show text-based τ -BENCH improvements by over 6% compared to strong RL baselines (using this mixed task and turn based rewards strategy). While the authors show various experiments with the text based system, they also experiment with multimodal models to show voice based interactions with the agents. The multimodal models improve by over 20% with the mixed training and comprehensive rewards strategy. Authors fine-tune Qwen3-8B and Qwen2.5-Omni-7B models and show improvements compared to strong baselines (PPO, GRPO and Reinforce-leave-one-out algorithms). Authors also conduct experiments to study how different reward strategies affect training performance as well as studying how incentivizing exploration involves both mixed task training and fine-grained reward schemes.
weaknesses: - the paper shows limited benchmark validations (primarily on the τ -BENCH Retail domain); the baseline experiments show Airline domain results are degrading generally and the main content of the paper does not focus on trying to show generalization improvements using their approach. (authors callout that their focus is on optimizing RL strategies for in-domain performance; however instead of focusing on analyzing and improving RL strategies for one set of results, this research is more applied where authors refocus on showing multimodal systems improvement)
- authors talk about this problem of reduced exploration with RL training, trying to understand how to recover from errors and building strategies for error recovery would be very interesting to analysis (what happens if judges give incorrect rewards, trying to understand these failure modes would make this work more interesting)
- it isn't clear how real and robust is the speech pipeline; SeeTTS based clean speech (without noise, accents, messy speech) isn't really tested and would further degrade the multimodal performance (already poor in a controlled sandbox environment compared to SOTA larger models)

---

confidence: 4
contribution: 4
initial_rating: 4
presentation: 2
questions: 1. How is the confidence paradox addressed by the trajectory level rewards proposed in this paper? Do you observe any qualitative or quantitative results indicating 
2. How did you verify that the LLM-as-a-judge was providing reliable answers on the turn level? 
3. From the findings of this paper, do you believe that turn-level rewards are viable future research direction and we simply need to engineer these rewards to be more useful and the credit assignment algorithm to be better or should practictioners and researchers focus on more useful trajectory-level rewards and mixed task training? 

This reviewer would be willing to give a higher score if the authors reframe the paper away from turn-level rewards, or some suitable explanation from the authors on why they chose to frame the paper in this way.
soundness: 2
strengths: 1. Introduce a sandbox environment combining speech and text in the tao bench environment. 
2. Perform multimodal training on speech and text and find that speech and text training using RL is a powerful tool for improving performance on both the text alone set up and the multimodal configuration of the environment
3. The paper presents a mixed-task training curriculum wherein the authors mix together math problems with tao agent and multi-modal rollouts. The paper shows that including these math tasks significantly improves performance, an interesting finding for the community.
4. The paper provides a detailed analysis of using a trajectory level reward from turn level reward metrics and demonstrates that using this can yield substantial improvements on tao bench. 
5. Ablate against other entropy based methods for turn level credit assignment and find them to not substantially improve performance on this domain. 
6. Formulate an LLM-as-a-judge reward assignment paradigm, though the details of how the LLM-as-a-judge is implemented are lacking
summary: This paper provides several interesting contributions to the community, namely: (1) a sandbox for text and speech training with RL and (2) ablations showing that training with *speech and text* with RL is more effective on both domains than training with text alone, (3) a method using trajectory level rewards designed by an LLM-as-a-judge annotating turn-level rewards that improves performance significantly on tao-bench, and (4) results demonstrating that mixed-task training and curriculum learning are an effective strategy for RL.
weaknesses: 1. The abstract, introduction & title of the algorithm all seem to indicate that turn-level (e.g. token level) reward assignment is critical to algorithm success. However, the most successful variation of this algorithm involves summing up all of the token level rewards and placing it at the end of the trajectory. I find it hard to believe that if we are summing up the reward and placing it at the end of the trajectory we are receiving the fine-grained, turn-level credit assignment claimed in the third bullet point of the introduction
2. The second critical component of this algorithm is the LLM-as-a-judge, but there is limited details on how we verify that the judgements made by the LLM-as-a-judge are truly useful other than the downstream RL performance
3. It seems that the LLM-as-a-judge requires a ground truth trajectory something that could be hard to find in many other domains (such as coding where there are many correct solutions)


Decision: Reject