Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Surprise, which quantifies the degree of changes in agents' environment, has received significant attention in the case of SARL but remains an open problem for MARL. This paper explores surprise minimization in MARL by utilizing the free energy across all agents. The formulation of the energy-based model is theoretically akin to the minimum conjugate entropy objective. They further validate the theoretical claims in several MARL benchmarks.
strengths_and_weaknesses: Strengths

1. **Importance of the problem**. The problem this paper considered is important. How agents behave in the presence of sudden environmental changes is critical in both SARL and MARL. And this paper step forwards this topic in MARL.
2. **The novelty of the proposed method**. The proposed method is novel in a theoretical view. They have deeply investigated the influence of the introduction of energy-based surprise minimization in MARL.



Weaknesses

1. **Quality of empirical evaluation**. Not adequate. (1) The compared algorithms, QMIX, VDN, COMA, and IQL, are not SOTA anymore. More new baselines are expected, such as RODE, and QPLEX. (2) There are other difficult scenarios of SC2, what are the results of EMIX in these scenarios?
2. **Significance of results**. Not significant. In Table 1 and Table 2, EMIX only archives incremental improvements over outdated baselines. 
3. **Clarity**. The writing need to be improved. For example, what does Figure 1 want to express? Why is the a superscript $a$ in $V_{surp}^a(s,u,\sigma)$, as there is no specific information related to agent $a$?
questions: Some questions are listed above, here are other questions.

1. In Eq. (3), Why is there still a superscript $a$ after summing over $a$?

2. How to learn $V_{surp}^a(s,u,\sigma)$?  What is the definition and implementation of state deviation $\sigma$? What does the black and white grid in Figure 2 mean?

3. Why do SC2 and Predator-Prey need surprise minimization?

4. Can the authors illustrate more about "minimum conjugate entropy objective"?
limitations: The authors have described the limitations of this work. But in the view of reviewer, the main limitation is that some benchmarks are not suitable for surprise minimization. Maybe the reviewer does not well understand surprise minimization, hope the authors can give more intuitive explanations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Paper introduces a new algorithm within the paradigm of Centralised Training and Decentralised Execution for multi-agent games. The authors introduce a new method EMIX - which improves upon QMIX by adding a global surprise minimisation objective to the centralised-value function training. The energy-based model (EBM) approximation is shown to provide stability during training through ablations. 


The paper provides strong theoretical justification that 1) the surprise operator converges to a fixed-point, and 2) that as the policy converges to an optimal policy then surprise is also minimised. The paper also demonstrates improved performance than baselines on most games within Starcraft II and through ablations demonstrates the necessity for both EBM approximations and surprise reward shaping.
strengths_and_weaknesses: Strengths:

Strong theoretical justification

Clear Presentation 

High quality paper

Seems like an original extension in the Centralised Training Decentralised Deployment algorithm family.

Weaknesses:

The global surprise encoder could be given slightly more description. The diagram is good but an additional example of the flow for interference or training optimisation would be appreciated. 

Performance is only marginally better  (a more challenging benchmark might be necessary). This is the most unconvincing component of the paper but i do appreciate the limited benchmarks available currently.   


questions: 1. Could the authors provide any more indepth analysis of how this outperforms COMA and QMIX, the analysis given in E1 is exceptionally useful and would be appreciated if more could be given of this flavour. I’d even recommend bringing more of this into the main paper.

2. The constant beta is not tuned at all during training - similar to SAC could you not train this?

3. The global state encoder and standard deviation encoders are a slight mystery to me and I’d like more understanding of how they are set up and trained.

4. It is unclear how the V_surp is learnt or obtained. What is the surprise method trained on, how is this estimated during training? 

limitations: The authors have discussed the limitations of this approach as it is scaled.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed incorporating surprise minimization in multi-agent reinforcement learning (MARL) through temporal energy models for capturing the uncertainty across agents. More specifically, this paper formulated an energy function extending the energy-based model theory and proposed an operator to temproally update this energy function. Moreover, this paper analysed the relationship to the soft Q-learning and gave a convergence analysis for the proposed method. With practical function approximation, the proposed algorithm according to the theory called EMIX was shown achieving superior performance with tiny margins compared with the baselines.
strengths_and_weaknesses: **Strengths:**
1. The presentation of this paper is good, starting with a reasonable question as the motivation and addresing it by a complete methodology.
2. The theoretical proofs are generally correct.
3. The related works discussed in this paper is sufficient.
4. It compared the relation to maximum entropy framework.

**Weaknesses:**
1. Assumption 2 is inconsistent with the discounted problem considered in this paper. To my best knowledge, there does not exist a termination state in the context of discounted problems. Nevertheless, it exists in the stochastic shortest path (SSP) problem.
2. The experimental results of baselines could be underestimated compared with the previous works.

questions: 1. Assumption 2 needs more explanation on why the termination state appears in the discounted problem settings.
2. I wish the authors give the details on how Eq. (17) is obtained.
limitations: The authors gave the limitations and the future work to address it.
review_title: null

Meta-review: metareview: All reviewers appreciated the quality of the paper, its contributions, clarity and theoretical justification, and novelty. I agree with the reviewers in recommending acceptance.
confidence: Certain
Decision: Accept