DPM: Dual Preferences-based Multi-Agent Reinforcement Learning

Authors: Sehyeok Kang, Yongsik Lee, Minu Kim, Jihwan Oh, Song Chong, Se-Young Yun
Primary area: reinforcement learning
Keywords: multi-agent reinforcement learning, preference-based reinforcement learning, RLAIF, RLHF

Abstract:
Preference-based Reinforcement Learning (PbRL), which optimizes reward functions using preference feedback, is a promising approach for environments where handcrafted reward modeling is challenging. Especially in sparse-reward environments, feedback-based reward modeling achieves notable performance gains by transforming sparse feedback signals into dense ones.
However, most PbRL research has primarily focused on single-agent environments, with limited attention to multi-agent environments.
In this paper, we propose Dual Preferences-based Multi-Agent Reinforcement Learning (DPM), which extends PbRL to multi-agent tasks by introducing _dual_ preferences comparing not only whole trajectories but also individual agent contributions during transitions. Furthermore, DPM replaces human preferences with those generated by LLMs to train the reward functions. Experimental results in the StarCraft Multi-Agent Challenge (SMAC) and SMACv2 environments demonstrate significant performance improvements over baselines, indicating the efficacy of DPM in optimizing individual reward functions and enhancing performances in sparse reward settings.

Decision: reject
Reviewer scores: 6, 5, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730451898651

thread_type:
official_review

<thread_object>
creator: reviewer_nvua

<content>
summary:
This paper addresses the sparse reward problem in cooperative MARL utilizing PbRL techniques. Specifically, besides comparing trajectories like SARL settings, this paper compares / ranks agents in a state to better learn the reward models. It also uses LLMs for automated comparison to reduce the reliance on human input, and achieves good results on benchmarks SMAC and SMACv2.

strengths:
- This paper is well written and easy to follow.
- The idea of extending PbRL into multi-agent settings by comparing / ranking agents is inspiring, which can be considered as an implicit credit assignment.
- Experiment results show that DPM can well “reconstruct” the dense rewards in SMAC scenarios, thus achieving comparable performance with the baseline that learns from original dense rewards.
- This paper provides an example of transforming SMAC game information into texts, which can be beneficial for relevant future research.

weaknesses:
- The reward calculation for running QMIX is not clearly illustrated. According to Line 307, intrinsic rewards are $\hat{r}$ and $\hat{R}\_t$, which are not defined in previous texts. I assume that $\hat{r} = \hat{R}\_t = \sum\_{j \in N} \hat{r}_{\psi} (s_t, a_t^j, o_t^j)$ according to Line 229. However, this reward function does not take $a_t^{-j}$ as input, which does not align with MARL settings where the rewards for individual actions depend on teammates’ actions.
- Even if the individual rewards are optimal, the sum operation will lose their information. For example, the reward sum is the same when individual rewards are (1, 0, …, 0) and (0, 0, …, 1). Authors should develop techniques to handle this problem, or apply DPM to MARL algorithms, like MAPPO[1], that can directly learn from individual rewards.
- There has already been a series of work[2,3,4,5] that utilizes LLMs to build dense rewards to solve sparse reward tasks. This paper should introduce these work and include them as baselines.
- StarCraftII is a popular game and SMAC is a popular benchmark. It is likely that LLMs already know the original dense reward function and use it for comparison. However, we should be more interested in whether DPM can solve novel sparse reward MARL tasks. It would be better if the authors discuss about this open problem.
- It seems unnecessary to include both versions of SMAC as benchmark, especially when the earlier version has “deterministic limitations”.
- More experiment details should be highlighted, including baseline introduction, the process of reward model learning, results with different LLMs, and the cost of LLM api for conducting the entire research and a single DPM running.

[1] Chao Yu, et al. The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games.

[2] Tianbao Xie, Siheng Zhao, et al. Text2Reward: Reward Shaping with Language Models for Reinforcement Learning.

[3] Yecheng Jason Ma, et al. Eureka: Human-Level Reward Design via Coding Large Language Models.

[4] Shengjie Sun, Runze Liu, et al. A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learning.

[5] Yun Qu, et al. Choices are More Important than Efforts: LLM Enables Efficient Multi-Agent Exploration.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730536851638

thread_type:
official_review

<thread_object>
creator: reviewer_yjrv

<content>
summary:
This paper proposes a preferences-based MARL algrorithm named DPM which aims at solving the sparse reward problems in MARL by utlising LLM to label trajectory preferences across trajectories and agent preferences within one time step. The preferences across agents help to identify agents' contribution within one step. The experimental results on SMAC and SMACv2 show that DPM outperform many strong baselines in sparse reward setting and achieve close performance to QMIX in dense reward setting.

strengths:
The paper is well written and most parts of the paper are easy to follow. The paper conducts a comprehensive set of experiments with ablation studies. The idea to access an agent's contribution to a team by utlising agents preference labelled by LLM is novel and the results demonstrate its effectiveness.

weaknesses:
Some essential details or results are missing, so I scored the paper's soundness as fair:

1. How long is a trajectory used in trajectory comparison. Is it a complete episode or part of the episode? If it is the former, how does it align with the extrinsic reward provided by the environment in sparse reward setting.

2.  Could you compare the DPM performance based on human feedback with LLM feedback on the same graph?

3. In line 344, you mention spaese-reward based algorithms perform poorly becasue of a harsher sparse setting, could you compare the sparse settings explicitly?

4. Could you analyse the reason that DPM outperforms QMIX dense in 10gen-terran? does it mean LLM can do better than extrinsic dense rewards?

5. Could you provide some actual examples of the LLM's prompt and cooresponding response on trajectories and agents comparison?

6. Do you think the results of Figure 8(d) contradicts with your motivation that we should assess the agent's individual contribution to the team for better performance? How can the trend that "feedback from only a subset of agents, especially with fewer agents, results in higher win rates compared to using feedback from all agents." support your motivation. Could you analyse the reason for the trend? Also, could you suggest a way to selecet the number of agents we should compare in large scale games and which agents should be compared? 

Some potential limitations are not discussed or addressed, so I scored the paper's contribution as fair:
7. How do you think the tradeoff between the cost (time and money) and the performance achieved by DPM? Given the same number of floating-point arithmetic calculations, will there be alternative apporach?
8. Does DPM rely on the prior knowledge of the games? In many sparse reward problems, even human experts may find it diffcult to compare agents' contribution within one step. 

I am willing to alter my socres if my questions are well adressed.

questions:
Please refer to my weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730621393739

thread_type:
official_review

<thread_object>
creator: reviewer_an6t

<content>
summary:
This paper addresses the challenge of sparse rewards in multi-agent reinforcement learning (MARL) by introducing Dual Preferences-based Multi-Agent Reinforcement Learning (DPM). DPM is a dual preferences-based method that evaluates both complete trajectories and individual agent contributions. Experimental results on the StarCraft Multi-Agent Challenge (SMAC) and SMACv2 demonstrate that DPM significantly outperforms existing sparse-reward baselines.

strengths:
1. The preference-based method appears to be novel in MARL settings and it is shown effective in the experiments.
2. The case study and ablation study section provides comprehensive insights.

weaknesses:
1. My primary concern lies in evaluation.

    - DPM is mainly evaluated against exploration methods in MARL. This comparison lacks fairness since DPM receives additional environment knowledge through LLM prompt templates (including game objectives, cues on how to win the game, and state interpretation) while baseline algorithms (except densely rewarded QMIX) do not have access to such information.
    - The evaluation relies solely on StarCraft-based benchmarks, lacking diversity. Additional benchmarks with different characteristics would strengthen the evaluation.

2. The motivation for agent ranking requires better justification. The direct comparison of agent actions in partially observable or heterogeneous settings (where agents have different types and abilities) needs more explanation.
    
3. The use of multiple reward functions introduces additional training overhead. Additionally, since preference pairs are generated from the replay buffer throughout the training, the computational costs of LLM queries should be reported.

4. Minor comments:
    - Line 179, typo with actions $o_t$

questions:
1. In the manuscripts, the author mentioned DPM adopts multiple reward models. How many distinct reward models does DPM employ? What are their specific roles (trajectory vs. agent preferences)? And how do these models differ in structure or function?
2. What is the required number of preference pairs per scenario? And how does the number of pairs affect performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650252107

thread_type:
official_review

<thread_object>
creator: reviewer_ypwc

<content>
summary:
This work introduces a preference learning method for the multi-agent RL setting. In particular, the approach, DPM, leverages traditional trajectory comparison feedback along with agent comparison preference feedback, which ranks agents' actions.

strengths:
1. The results in SMAC and SMAC v2 show that adding DPM can significantly outperform other standard MARL baselines.  

2. I liked how the authors included additional generalization experiments, demonstrating that DPM can work well with other MARL algorithms, not just QMIX.

weaknesses:
1. Unclear what the contributions are / limited novelty:

This method's significant contribution seems to be its use of a new feedback methodology, where a teacher ranks specific actions. 

2. Experimental Design:

The authors did compare DPM against several MARL baselines but not against the other preference learning algorithm for multi-agent systems [1]. I think the authors need to compare against this algorithm or explain why this would not be a valid comparison.

The evaluation is limited to one environment domain, SMAC. Why not include other prominent MARL environment suites, such as Multi Particle Environments (MPE) or Multi-Agent Mujoco?

3. Use of LLM as preferences:

The authors use LLMs to obtain preferences, which is a design choice. However, the reasoning for this design choice is limited. The authors note that DPM uses preferences from LLM to address the issues associated with human preferences. Can the authors elaborate on these issues? In addition, the authors should describe the limitations of using an LLM in this context.  

We should also note that one primary objective in preference learning is to learn reward functions that encode human preferences (i.e., it is important that humans are actually providing the feedback). Can DPM work with human preferences? I’m a bit concerned that humans would have difficulty ranking specific actions in the agent comparison component of the approach. Especially in the continuous action setting, actions from one time step to the next are unlikely to change drastically, making it difficult for a human to detect differences. 


[1] https://ojs.aaai.org/index.php/AAAI/article/view/29666

questions:
1. How much preference feedback is being used in Figure 4? Is the same amount of the two types of preference feedback being used?

2. In the preference alignment case study, it is unclear what is being aligned. Can the authors elaborate on what is being compared?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730768037637

thread_type:
official_review

<thread_object>
creator: reviewer_jjs9

<content>
summary:
In this work, the authors propose generating agent-based intrinsic reward functions in MARL whose functions are generated by combining a preference score based both on pairwise trajectory comparison as well as a comparison between agent decisions. The use an LLM approach to generate the preference comparisons and compare on both SMAC and SMACv2. The contribution is simple, but interesting and the paper explores many different comparisons which is well done.

The exposition and writing can be improved as well as small mistakes such as Figure 1. They compare against many baselines, but in a completely sparse reward setting where as the baselines were developed with a ”weakly” sparse reward which perhaps makes the comparisons unfair. Further, they have omitted some recent work in Sparse Reward MARL - see below. Additionally, only SMAC and SMACv2 are compared, but other MARL environments would be interesting to add - although, in this reviewer’s opinion are not necessary. Additionally, the time constraint of training through an LLM is only very briefly mentioned at the end and perhaps further analysis can be conducted on the interpretability of their learned reward functions compared to the dense human version.

strengths:
*I strongly appreciate Figure 2.

*For feedback, they use an LLM for feedback of the RL agent. For completeness, the limitations of such a choice must be at least mentioned in the introduction, but I enjoyed the comparison with Human feedback in Appendix D.2 and preference Alignment.

*The paper structure overall is very clear and the idea is simple but intuitive.

*The ablation study is quite nice. I appreciated this as well.

weaknesses:
* Figure 1b has no blue line or perhaps it is always zero of which it is useless and hard to see.
* Figure 1a is extremely confusing to follow.
* There is a real lack of related work section compared to what has been done in MARL on sparse versus dense reward functions. No details about the ideas the baselines have introduced in the sparse MARL space are discussed.
• What is Ψ in P_ψ, \hat{r}_ψ for example in equation (1) - I assume it is the learnable weights, but this should be stated.
* Probably should add that the loss is negative log-liklihood in Eqn. (2) as other approaches can be used
* In SMAC, it is very challenging to win with the completely sparse approach. Thus, I think a comparison with the ”less weak” sparse manner is more fair as I imagine the baselines will do better, but the results of DPM+QMIX will not change much. This ”less weak” sparse reward is very natural to construct of which a non-expert could reasonably create. Comparing for example with ”https://arxiv.org/pdf/2405.18110”, it is clear that for 5m v.s. 6m, the result is extremely similar in terms of performance if one uses their weakly sparse reward
* A comparison with the following sparse reward solution to SMAC in ICML 2023 should at least be mentioned or discussed. (”Lazy Agents: A New Perspective on Solving Sparse Reward Problem in Multi-agent Reinforcement Learning”)
* There are a few typos and the flow of the paper is incoherent at times. (End of Section 2.1 for example). Shortened List of typos:
– The title should be singular. "Dual Preference Based MARL"
– Section 2.1, second paragraph. Third sentence should be "in which subgoal-based..."
– Section 2.1, second paragraph. Fourth sentence should be "caused by"
– Section 2.2, 3rd, sentence "a scripted teacher which assigns"
– Fix the bibliography. Many things such as ”ai”, "Smacv2" etc. should be capitalized.
– Please use "LLM" as a singular vs plural correctly. For example in first pargarph of figure 3.1, one needs the article "an" in many places.
– Section 3.3., "common PbRL research"

questions:
* The relative size of L^T versus L^A may not be the same as one is summing many more terms in (6). Is this addressed with some normalization?
* Is it fair to compare with sparse reward scenarios? I ask because essentially, the approach is building its own dense reward function via trajectory and agent preferences and so perhaps it should be compared from a dense perspective.
* The temporal component of training through an LLM is not discussed (somewhat discussed in the conclusion, but not explicitly). How does it compare the computation of a densely constructed rewards as one needs to evaluate a NN for each agents intrinsic reward. This may scale badly as the number of agents become large compared to human constructed reward functions although subsets may be used as mentioned.
* It would be interesting (although not necessary) to do an analysis comparing the reward generated via DPM vs the dense reward for the problem. Perhaps the DPM generated reward has some sense of interpretability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732090532193

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response

comment:
### **5. Revision of Main Text and Figures**
Thank you for pointing out the typos. we have corrected the mentioned typos as well as other typos throughout the paper. Revisions have been highlighted in blue in the text. 

### **6. Additional experiments in a weak sparse reward environment**
Reviewer Jjs9, an6t and YJrV raised questions about experiments under weak sparse reward setting. Many researches have adopted weak sparse reward settings and to support your evaluation of our approach, we conducted additional experiments comparing the performance of our method with the baselines in a weak sparse setting(Figure 10, lines 902-909). Since our method is not significantly influenced by extrinsic rewards, we evaluated it in a hard reward setting, and the results demonstrate that our method outperforms the baselines even in more extreme environments. Additionally, we compared our approach with the method mentioned in the paper reviewer Jjs9 referenced(LazyAgent). In the case of LazyAgent, additional information must be proivded, requiring human experts to anlyze observations and categorize them appropriately, making it difficult to consider this a completely fair comparison under identical conditions. Despite this, our method consistently shows better performance across various scenarios.

---
[1] Zhu, Tianchen, et al. "Decoding Global Preferences: Temporal and Cooperative Dependency Modeling in Multi-Agent Preference-Based Reinforcement Learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 15. 2024.

[2] Klissarov, Martin, et al. "Motif: Intrinsic motivation from artificial intelligence feedback." arXiv preprint arXiv:2310.00166 (2023).

[3] Warnell, Garrett, et al. "Deep tamer: Interactive agent shaping in high-dimensional state spaces." Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.

[4] Lee, Kimin, Laura Smith, and Pieter Abbeel. "Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training." arXiv preprint arXiv:2106.05091 (2021).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732090602879

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response

comment:
## **Common Response**
Dear reviewers, we would like to express our gratitude for your invaluable feedback on our paper. In response to the reviewers’ comments, we provide our common our answer below. We kindly ask the reviewers to refer to the updated manuscript.

---
### **1. Appropriateness of Experimental Setup (Fairness)**
Reviewer Jjs9, an6t, and YJrV raised concerns about whether it is fair to compare the proposed approach with sparse reward-based MARL algorithms. Since PbRL provides additional information in the form of preference feedback compared to general RL algorithms, it is preferable to compare PbRL algorithms against other PbRL approaches for a completely fair comparison. However, while PbRL has been extensively studied in single-agent RL, there is a lack of research in the MARL domain. 
Although there is a related study (MAPT [1]) in the offline RL domain presented at AAAI2024, comparing it with our research would not be fair as it operates in an offline RL setting, utilizing expert trajectories and tens of thousands of preference feedbacks to train a reward model. Therefore, to establish baselines, we referenced the initial studies on PbRL and RLAIF to determine our setup. 

MOTIF [2], as the first study to leverage LLMs for feedback, faced the challenge of having no comparable approaches and thus set RND, an exploration method known for sparse reward environments, as the baseline. Inspired by this, we set algorithms for sparse reward environments as our baseline. Furthermore, 'Trajectory' in Figure 5 can be considered as basic PbRL algorithm in single-agent RL.
Additionally, researches such as DeepTAMER [3] and PEBBLE [4] compared the performance of PbRL methods with general RL algorithms (e.g., SAC, PPO) in dense reward settings. Similarly, we used QMIX as the baseline in dense reward settings.

In summary, since there is no prior research applying PbRL to MARL, a perfect basline setting is limited. However, we conducted experiments to evaluate our method from various perspectives, and we believe these experiments will be helpful in evaluating DPM. Finally, we believe our study is a pioneering effort in PbRL research within MARL, laying the foundation for future research and serving as a milestone in this field.

### **2. Performance Comparison in Cases Using Human Preference**
Reviewer yPwC and YJrV raised concerns about whether DPM can utilize human feedback for learning. Of course, DPM is capable of learning based on human feedback. Figure 7 (lines 473-479) illustrates the performance of DPM when human feedback is used. As shown, DPM performs well when it utilizes both trajectory and agent action preferences, whereas it struggles in scenarios where this is not the case. The process of collecting human feedback is explained in detail in Appendix D.2 (lines 1015-1025). 

### **3. Regarding Prior Knowledge (LLM)**
Reviewer NVua and YJrV raised questions about prior knowledge of LLMs. Large Language Models (LLMs) are trained on vast amounts of data during pretraining, making it highly likely that they possess prior knowledge about the problems we aim to solve, including SMAC. However, for models like GPT, where training data and details are not disclosed, it is impossible to verify whether specific prior knowledge has been learned. To address this, we conducted experiments by modifying the prompts through prompt engineering to minimize prior knowledge as much as possible. We conducted experiments by removing SMAC-related information from the prompt, as shown in Figure 12 (line 918-944), and the results are presented in Figure 13 (line 948-959). The LLM is able to generate proper feedback even in the absence of explicit SMAC-related information. Moreover, in the 2m_vs_1z scenario, the performance was better when SMAC information was excluded. This suggests that while the LLM may possess prior knowledge about SMAC, such knowledge does not always contribute positively to generating feedback.

### **4. Alignment with Extrinsic Rewards or Human Preference**
Reviewer yPwC and YJrV raised questions about alignment with extrinsic rewards or human preference. We calculated the Pearson correlation coefficient (PCC) to compare the estimated reward produced by the reward function with the ground truth reward, as shown in Figure 9 (lines 843–900). The PCC was highest at 0.7 when both types of feedback were used, while it was lowest when only trajectory comparison feedback was used. Additionally, to evaluate how well the model aligns with actual LLM feedback, human feedback, and scripted teacher feedback, we measured the accuracy using 100 test trajectory pairs. Similarly, the highest accuracy was observed when both types of trajectory comparison feedback were utilized.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The reviewers acknowledged that the paper tackles an important problem setting of preference-based RL for multi-agent systems, and proposes an interesting approach that accounts for the comparison of individual agent actions along with the comparison of overall trajectory. However, the reviewers pointed out several weaknesses and shared concerns related to limited experimental evaluation, which lacks diverse environments and suitable baselines. We want to thank the authors for their detailed responses. Based on the raised concerns and follow-up discussions, unfortunately, the final decision is a rejection. Nevertheless, this is exciting and potentially impactful work, and we encourage the authors to incorporate the reviewers' feedback when preparing a future revision of the paper.