Title: ARIA: Training Language Agents with Intention-driven Reward Aggregation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Language Agent

Agent Learning

Reinforcement Learning
Abstract: Large language models (LLMs) have enabled agents to perform complex reasoning and decision-making through free-form language interactions. However, in open-ended language action environments (e.g., negotiation or question-asking games), the action space can be formulated as a joint distribution over tokens, resulting in an extremely large and combinatorial action space. Sampling actions in such a space can lead to extreme reward sparsity, which brings large reward variance, hindering effective reinforcement learning (RL). To address this, we propose **ARIA**, a method that **A**ggregates **R**ewards in **I**ntention space to enable efficient and effective language **A**gents training. ARIA aims to project natural language actions from the high-dimensional joint token distribution space into a low-dimensional intention space, where semantically similar actions are clustered and assigned shared rewards. This intention-aware reward aggregation reduces reward variance by densifying reward signals, fostering efficient and effective policy optimization. Extensive experiments demonstrate that ARIA not only significantly reduces gradient variance, but also delivers substantial performance gains of average 9.95% across four downstream tasks (e.g., negotiation and text-based games), consistently outperforming strong offline and online RL baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper seeks to improve the reward sparsity problem in RL by clustering high-dimensional actions together to share rewards. They do this clustering by using an embedding model on LLM actions and have a way of using the change in aggregated reward to determine how many clusters to use. There is some simple theory on how aggregating actions can improve variance of advantages and policy gradients. Empirical results show solid improvements on Twenty Questions, Guess my city, and some multi-agent scenarios like bargaining and negotiation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Interesting and intuitive method for dealing with the reward sparsity problem in high dimensional freeform text environments. There could potentially be extensions to other environments. \n- Solid empirical results on a variety of problems\n- Some positive results on extensions to iterative offline RL and to online RL\n- Clever method for determining the number of clusters for aggregation\n\nWeaknesses\n- The theory is overall not very insightful to me. For example Theorem 4.2 depends on the very strong assumption in definition 1, where actions after clustering are similar in both rewards and policy distribution - here there is not much lost from clustering and so the bias of the algorithm is small. I'm not sure what happens in practice - is there an empirical way to examine the bias?\n- It's not clear why the particular baselines were chosen - there was not much context here on why these are the strongest alternatives.\n- Some ablations to understand how the performance varies with the different choices made during clustering (how many clusters, which embedding model used, etc) could robustify the results and provide more understanding. There is a short ablation in Appendix B but not much insight can be gained from it other than that the number of clusters is a hyperparameter to set with a sweet spot.\n\nQUESTIONS:\n- would the proposed method work on the standard math reasoning datasets?\n- Qwen 1.5B in Table 3 doesn't seem to get a reasonable performance on Bargaining for either the baseline or ARIA?\n- how would you extend this method to other types of environments?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses challenges in open-ended multi-turn dialogue tasks, which require decision making to achieve a certain goal. The challenges arise from sparse rewards due to large action spaces, hindering the learning process. To address this, authors embed actions into an embedding space, and perform iterative clustering, finding an optimal number of clusters. Then, they average the rewards within the cluster and use it as an advantage in REINFORCE. They theoretically showed their approach decreases variance in both rewards and the policy gradients. Their experiments include extensive comparisons of modern online and offline methods, and they showed big gains in games benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n# Quality\n\n* (+) The paper provided theoretical justifications and extensive evaluation of their method, using multiple LLMs and benchmarks.\n* (+) Authors used proper statistical tests to compare against the baseline, and included corresponding p-values.\n\n# Clarity\n\n* (+) Paper uses consistent notation, making it easy to follow.\n\n# Significance\n\n* (+) Their results are strong across benchmarks and outperform SOTA methods by wide margins. \n* (-) Their approach might be sensitive to the choice of embeddings, which is listed in the limitation. However, it would still be useful to have some experiments using other embedding models. This could help with adoption.\n* (-) Not clear how this method would interplay with other variance reduction, like using actor-critic, or PPO-style losses. Perhaps, including this in a discussion section, or some suggestions for future work would make results stronger.\n\n# Originality\n\n* (+) Reward aggregation in a latent space is a neat way of reducing variance in learning.\n\nQUESTIONS:\nThank you for your work! I enjoyed reading your paper.\n\nMy suggestions/questions are:\n\n* Figure 4.a, before & after aggregation have the same distribution in the bottom two plots. Why do you think that is? I understand the top two plots had a big variance and it was reduced by using aggregation.\n* It would be helpful to quantify the effect of embeddings on your method. One way could be to try ablations using other embedding APIs.\n* It would also be useful to include visualization of embeddings and resulting clusters for some of the samples in your benchmarks.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper proposes an intention-driven reward aggregation method to train agents on open-ended agentic tasks. The method helps to reduce the variance of advantage and gradient, therefore aiming to enhance the stability and performance of agent training.

STRENGTHS AND WEAKNESSES:
### Strengths

1. The paper is well-written and easy-to-follow.
2. The proposed reward aggregation method is interesting and novel, supported by theoretical analysis.
3. The experiments are comprehensive and convincing.

### Weaknesses

1. The proposed method maps the natural language open-form action to a low-dimensional intent space and aggregate rewards based on the clustering results. It could encourage the model to learn to select better intentions. However, in open-ended agentic tasks like bargaining, the language tactics, sentence formation, precise use of language, and stylistic choices are also important. And I wonder if these factors are considered during the clustering process. Overall, there are other important skills to learn besides *intention*. If the mapping excludes the other factors, will the resulting agent performance limited by the training algorithm (like it can produce the right intention but fall short on other aspects) ?
2. In the experiments, it seems that the model continues to improve from 1 to 3 iterations in some tasks. It would be beneficial to provide results with more iterations and some analysis to understanding the ceiling of the method.
3. I think it would be interesting to evaluate the models on some out-of-domain tasks to see if the agentic capability can be generalized to other environments, like some discrete action space environments.

QUESTIONS:
See in the weakness section.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method to improve the ability of language agents in open-ended environments. Specifically, they highlight that RL struggles in this setting due to reward sparsity and high gradient variance. They address this by proposing to project actions from a high-dimensional joint token distribution space into a low-dimensional intention space, where semantically similar actions are clustered and assigned shared rewards. This is done using semantic embeddings and hierarchical clustering, resulting in denser reward signals and more stable optimization. They evaluate on four benchmark tasks (20 Questions, Guess My City, Bargaining, Negotiation) on both offline and online RL methods, improving on state of the art methods by 10%.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The authors provide adequate motivation for why this is a challenging problem due to reward sparsity and high variance in open-ended language RL\n- There is sufficient theoretical justification and analysis of their methodology\n- Empirical results are solid, and the inclusion of both single-agent and adversarial tasks gives the method a reasonable range of applicability\n\nWeaknesses:\n- While the online variant is tested in single-agent games, there is no evaluation of online ARIA in adversarial tasks, which would help generalize the method's scalability in more dynamic environments\n- Specifically for the task of negotiation, there exist several other dialogue tasks, and I am not convinced that intentions can be separated in more complex negotiation environments, where wording choice really changes the dynamics of the negotiation. I would have liked to see these tested\n- The clustering process assumes discrete intention boundaries, which may not hold in realistic multi-goal or multi-agent dialogues. It is unclear how well this assumption scales when linguistic nuances and pragmatics matter\n- The method relies heavily on the quality of sentence embeddings, but there's limited discussion on how embedding choice or drift over time might affect performance.  Poor embedding quality or inappropriate granularity may lead to underfitting or over-smoothing. The authors address this with SplitScore, but it's unclear how robust it is to semantic edge cases or in multi-goal settings\n- The authors could benefit from a more thorough literature review and motivation on benchmarks for this task\n\nQUESTIONS:\n1. How does ARIA handle semantically overlapping or hierarchical intentions, where actions might belong to multiple latent goals?\n2. Do the authors plan to extend their online training experiments to two-agent tasks, where strategies evolve more dynamically?\n3. How would ARIA handle concept drift in natural language use, for example, due to changing user preferences?\n4. Does hierarchical clustering not become a bottleneck for very large-scale training?\n\nLIMITATIONS:\nYes, they have addressed Limitations in Appendix, Section A, where they discuss that their method relies on clustering in the semantic embedding space to define intention groups, which they note addresses two limitations. I think the second limitation stands out and might be noted as a weakness. For more complex tasks, it is hard to assume that intentions are discrete and can be separated, such as negotiation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}