Kati8/ICLR_2023_2025 huggingface dataset:
authors: Zhitao He
authors: Zijun Liu
authors: Peng Li
authors: Yi Fung
authors: Ming Yan
authors: Ji Zhang
authors: Fei Huang
authors: Yang Liu
reviews: 
Comments: This paper introduces a two-stage multi-agent learning framework to address the poor generalization and sparse reward issues in large language model (LLM)-based agent systems. In the first stage, the authors propose a data synthesis pipeline to automatically generate training data for fine-tuning the LLM. In the second stage, they leverage a critic agent to allocate rewards to each agent at every conversation round.
Strengths: 1. This paper studies an important problem: the generalization of cross-platform.
2. The idea of assessing the contributions of each agent is interesting.
Weaknesses: 1. The organization of the paper needs some improvement, especially MARL with Edge Updates Section.
2. The main concern is that the proposed Process Reward Decomposition is not well-justified: I am skeptical about using a critic agent to directly generate temporal and structural credit assignments.
3. Lack of comparison with related work on multi-agent LLM[a].


a. Zhuge, Mingchen, et al. "Language agents as optimizable graphs." arXiv preprint arXiv:2402.16823 (2024).
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: This paper addresses the deficiencies in flexibility and generalization in existing multi-agent LLMs and proposes the CollabUIAgents framework based on general environment knowledge learning and MARL in two stages. This enhances performance in Open-Source LLMs. However, the paper still needs improvement in specific expression, motivation, and writing style, as detailed below.
Strengths: This paper investigates an important direction in multiagent systems and poses a promising direction for future research.
Weaknesses: 1. In Task Formulation, it is necessary to clarify the concepts of  `agent` and `policy`. Furthermore, there needs to be consistency between a_t in Eq(3) and Eq(1). It is unclear whether a_t represents an action by a single agent, a joint action by multiagent, or an aggregation action, which should be specified.

2. In stage 1, why is the curriculum divided into three parts? or what is the insight?  It's better to explain the rationale behind choosing these three particular parts for the curriculum, and how each part contributes to the overall learning process.

3. How to comprehend  ''The rationale is that, for the critic agent, it might be more simple to identify whether a single decision is wrong, than to judge the reward of long decision chains between multiple agents''? It seems to contradict RL research. Instead, single-step rewards are usually not as accurate as long-term rewards, From the internal logic, judging a single-step decision is more difficult than multi-step.  It's better to provide evidence or reasoning to support their claim, especially in light of existing RL literature that suggests otherwise.

4. Multi-agent reinforcement learning methods are usually based on mathematical models such as Markov Game and Dec-POMDP. In stage 2, what is the multi-agent model, which needs a detailed explanation? How does it relate to or differ from standard models like Markov Games or Dec-POMDPs?

5. The nature of their reward decomposition process in stage 2  should be explained clearly. Is it artificial prior or adaptive learning? If it is artificial prior, it is no different from role allocation in form, and it is difficult to reflect the advantage of adaptive learning. How does it differ from or improve upon traditional role allocation methods?

6. In stage 2, it seems that each agent learns independently. How to ensure multi-agent collaboration? What are the mechanisms or techniques to promote collaboration between agents during the learning process in stage 2?

7. The paper also has several unclear character definitions and grammatical problems, such as：

    (1) It should be "each agent \pi_i” instead of  "all agents \pi_i” in line 271

    (2) It should be "the number of agents” instead of all "the number agents" in line 132

    (3) In Task Formulation, transition function and the maximum step are both represented by the character `T’.

    The reviewer suggests that the authors conduct a thorough proofreading of the paper, paying particular attention to consistency in mathematical notation and grammatical correctness.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: This paper presents the CollabUIAgents framework, which aims to improve multi-agent learning capabilities in real-world interactive environments, particularly addressing issues related to sparse rewards. By employing a two-stage learning process—general environmental knowledge adaptation and multi-agent reinforcement learning with process reward decomposition—the framework enhances adaptability and cross-task generalization.
Strengths: The method effectively tackles the challenges of sparse rewards and rigid role assignments in multi-agent learning. The two-stage learning process not only enhances adaptability across various tasks and environments but also allows for automated data generation, reducing the need for manual annotation. Furthermore, the comprehensive experimental design, including extensive comparisons with state-of-the-art models and thorough ablation studies, provides strong empirical support for the framework's effectiveness. The results indicate impressive performance improvements, demonstrating the framework's potential to advance multi-agent systems in real-world interactive scenarios.
Weaknesses: The experimental setup in this paper lacks clarity, particularly regarding the configuration and evaluation of different agent setups, which impacts the transparency and reproducibility of the experiments. Additionally, the framework is demonstrated with a relatively small number of agents, raising scalability concerns. As the number of agents grows, managing an undirected communication graph may become computationally expensive, potentially affecting performance. Furthermore, some figures could benefit from clearer annotations and layout to better convey the framework’s structure and processes.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3