Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focus on addressing the problem of high reward uncertainty in multi-agent reinforcement learning (MARL). To address this problem, this paper proposes a new Distributional Reward Estimation (DRE) framework, which is composed of multi-action-branch reward estimation and policy-weighted reward aggregation for stable training. The full method, DRE-MARL is built on top with the architecture of centralized training and decentralized evaluation (CTDE), which consists of N decentralized actors and a centralized critic. Empirically, the proposed method is evaluated in several MARL environments, i.e. cooperative navigation, reference, and treasure collection with different number of agents. Compared with other SOTA MARL methods, DRE-MARL achives the best performance in most of the environments using three types of reward settings. 
strengths_and_weaknesses: ==Originality==

The proposed DRE-MARL is novel and interesting. The main idea is to develop distributional reward estimation followed by policy-weighted reward aggregation for MARL. This idea is intuitively similar to human's decision making process that considers all possible consequences of all action branches. The connections and differences of this work and previous work are well discussed and the related work are cited in the paper. 


==Quality==

The proposed method is technically sound. The experiments are conducted in diverse settings with ablations, and the results well supports the claims. Although there are some limitations, this work is a complete piece of work in addressing the reward uncertainty problem in MARL.  

==Clarity==

This paper is clearly written and well organized. The problem is well formulated with a clear intruduction of the cause of reward uncertainty. The method is clearly motivated and introduced. The limitations and potential impacts are also discussed. 

==Significance==

This paper provides a novel distributional reward estimation method to address the high reward uncertainty in multi-agent reinforcement learning. The proposed method is novel and can be applied in other MARL methods. 

questions: 1. Line 55, "If we can obtain the potential rewards on other action branches, we will perform more stable critic updating and thus achieve better performance." Is there any assumption of requirement on the design of action branches to make this sentence correct? What if the other actions are opposite the the current action of a specifict agent? 

2. What is the motivation for the design of different aggregation weights?

3. DRE-MARL requires individual reward estimator for each agent. How muc is the computational cost gain of DRE-MARL compared with other SOTA-MARL methods? 
limitations: The authors have discussed the limitations and broader impacts of their work in section 7. 

==Limitations==
1. DRE-MARL requires a prior assumption about the form of reward distribution. Although the distribution usually can be chosen arbitrarily, it may hurt performance when we choose a very complex distribution.
2. Besides, reward aggregation is only used in discrete action space. If would be better if the method can be extented to more general continuous action space. 

==Broader impacts==
The authors does not see any negative societal impacts of this work while using the proposed method in practice.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Reward uncertainty is a longstanding problem in multi-agent reinforcement learning, which stems from two aspects: the natural uncertainty in the MDP environment, and the actions of other agents. To deal with the reward uncertainty problem, this paper proposes a new framework for estimating the reward distribution and aggregating the estimated reward distribution. Then the proposed method uses the aggregated rewards to update the centralized critic network and the decentralized actor networks. Experiments in the particle domains demonstrate the effectiveness of the proposed method.
strengths_and_weaknesses: To my best knowledge, the idea of estimating the reward distribution and using the aggregated rewards to optimize the policy is novel, but I get no sense how significant reward estimation is in the multi-agent reinforcement learning domain. Ablation studies on removing the reward estimation module could answer this doubt.

The writing of this paper is mostly clear. In addition, this paper has conducted thorough comparative experiments to compare with state-of-the-art multi-agent algorithms in the particle experiment domain. A more challenging multi-agent benchmark is Starcraft 2. The authors are encouraged to do more experiments in this domain.

questions: Section 6.3 has conducted ablation studies on different reward estimation methods. I wonder what will happen when no reward estimation is conducted (just using the external rewards to update actor and critic networks). This experiment could better demonstrate the utility of reward estimation in multi-agent reinforcement learning.

This paper refers to the proposed framework as “two-stage” learning (Line 162). Is the reward estimation conducted before the reward estimation? If so, where does the data for estimating the reward distribution come from? Or those two stages are conducted concurrently? This point should be clarified. 

Why is the regularization term $L_R$ needed in the reward estimation objective？ What if this term is removed? 

limitations: The authors have discussed the possible limitations of their work in the last paragraph of the submission.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Considering that high reward uncertainty remains a problem, the authors propose a novel distributed reward estimation framework to enhance multi-gent reinforcement learning. The training process is stabilized by designing multi-action-branch reward estimation and policy-weighted reward aggregation. Multi-action branch reward estimation is first employed to model the reward distribution of all action branches, and then reward aggregation is used to obtain a stable update signal during training. And its effectiveness is demonstrated experimentally.
strengths_and_weaknesses: Strengths.
* paper writing is well understood
* The pictures drawn can help understanding
* The motivation is clear and seems to make sense
* table-1 looks like a lot of experiments were done
* Ablation experiments also look interesting


Weaknesses.
* Lack of discussion and experimental comparison with work related to Distributional MARL

questions: * Shouldn't this work be discussed and compared with the related work of Distributional MARL? For example. 
    * DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning
    * A Distributional Perspective on Value Function Factorization Methods for Multi-Agent Reinforcement Learning
    * RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents

limitations: None
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper outlines a new method for multi-agent reinforcement learning settings (DRE-MARL) which primarily focuses on reward estimation for multi-agent reinforcement learning settings. The authors first describe the problem settings and motivate why reward estimation is important and challenging in multi-agent reinforcement learning and how DRE-MARL differs from prior approaches, including reward uncertainty method and reward estimation methods in single-agent reinforcement learning. According to the authors' summary, many reward estimation methods struggle in multi-agent setting due to the additional complexity and greater sources of uncertainty. The authors identify mutual interaction between agents and natural disturbance of the environment as two sources of uncertainty in MARL settings and propose a method simulate reward uncertainty by applying stochastic processes.

Subsequently the authors outline their method (DRE-MARL) for reward estimation based on a distribution of potential action a single agent can take in a MARL setting and also describe different reward aggregation methods that are tested in the experiments. The authors then compare their method against different MARL baselines across a set of collaborative MARL environments showing mostly outperformance and conduct an ablation study for various reward aggregation schemes.
strengths_and_weaknesses: **Originality:**
- Strengths: The paper proposes a new method for reward estimation in MARL settings that differs from prior approaches and shows good performance compared to baseline methods. Relevant work is cited and compared against in the beginning of the paper.
- Weaknesses: The authors could have provided more detail on the limitations of their method (discussed below as well)

**Quality:**
- Strengths: The methods and contributions are generally well supported in the experiments and discussed in detail in the paper.

**Clarity:**
- Strengths: The paper is generally well written and well organized with relevant equations, diagrams and descriptions.
- Weaknesses: The authors could have made better use of figure captions (in Figure 2 and Figure 3 specifically) to make it easier for the reader to understand the relevant messages conveyed by the figures.

**Significance:**
- Strengths: The paper proposes a new method in a relevant subject area of MARL.
- Weaknesses: The authors could have provided more detail on the limitations of their method and put it into the broader context of MARL methods (discussed below as well).



questions: **Nits**
- A is not capitalized in title "Multi-agent"
- first page says 35th Neurips, which is 2021
- Line 173-175 (page 5) have some spelling errors ("interplay grows exponentially with an increase of the agent number")

**General Questions:**
- Could you clarify why you choose a centralized critic for your method? Is this related to the environment setting (i.e. the environment only provides a single reward for all agents)? Can you see your method working with a de-centralized critic? what would change if anything?
- Could you say more about how your method might perform in competitive MARL settings? Right now you have looked at cooperative settings and it would be interesting to contrast that with competitive settings. (No new experiments needed, mainly looking for additional detail)
- It seems like p2p-MARL is the most competitive method to DRE-MARL and also uses reward estimation. Could you clarify the differences between p2p-MARL and DRE-MARL?
- How did you choose k for the distributional estimation? Was there a significant difference between different values of k?
limitations: I think that the paper could be improved by further discussion of the limitations of DRE-MARL. As far as I can tell only one limitation is briefly mentioned on Page 9, which is that DRE-MARL and reward aggregation are only used in discrete action spaces. It would be good to get further clarity on the following from the authors:
- How could DRE-MARL be applied to continuous action spaces?
- What limitations does DRE-MARL have that can inspire future work?
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: this paper propose distributional reward estimation for multi-agent reinforcement learning (DRE-MARL). this paper focuses on the problem of reward uncertainty in MARL. The main idea of this paper is to design the multi-action-branch  reward estimation and policy-weighted reward aggregation for stabilized training. This former part is simply function approximation with historical data, while the latter part is weighted reward aggregation. Experiments show that DRE-MARL outperforms other SoTA algorithms comprehensively.
strengths_and_weaknesses: Strengths:

it is novel to solve the reward uncertainty problem in MARL

policy-weighted reward aggregation enables stable training of the critic and actors and it is quite robust. 


questions: I suggest the authors put the related work in section 1 introduction to section 2. 

I think the data volume for reward estimator should be very large, so I suggest the authors conducting some experiments to test how much data should be used. 
limitations: the reward distribution in reward estimation is hard to choose
review_title: null

Meta-review: metareview: The reviewers carefully analyzed this work and agreed that the topics investigated in this paper are important and relevant to the field. They believe that the NeurIPS community could benefit from the ideas and techniques presented in this work. They argued, e.g., that the paper is novel and interesting, technically sound, clearly written, and that the method is clearly motivated and introduced. One reviewer expressed a few technical concerns, to which the authors responded appropriately. The authors have also, post-submission, further compared their model and other baselines from different perspectives. One reviewer pointed out that a limitation of the paper is the lack of discussion and experimental comparison with other work related to Distributional MARL. The authors responded to this, but the reviewer requested further details and a more thorough discussion; the authors then expanded their initial response via two detailed rebuttal messages, which were considered to be satisfactory. Finally, another reviewer (who also expressed positive views on this work) mentioned that the authors could have provided more details on the limitations of their method. Overall, all reviewers were positively impressed with the quality of this work and look forward to an updated version of the paper that addresses the suggestions mentioned in their reviews.
confidence: Certain
Decision: Accept