Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method in the setting of centralized training and decentralized execution for training policies which minimize surprise.  The method uses insights from energy based models, deriving an energy operator that forms a contraction operator on the appropriate value functions, and uses this to make a variant of QMIX, which promotes surprise minimization across the multi-agent system.
main_review: This work builds in the interesting new directions around surprise minimization and energy based models.  Moreover, it provides a powerful theoretical framework incorporating ideas from energy based models and providing an algorithm with convergence grantees.  To the best of my knowledge the mathematical results seem correct, though I have not checked the work in the Appendix, they are intuitive and seem to be extensions of existing ideas.  

Though the theoretical results are strong, the empirical results are not as compelling.  For an evaluation of this method, I would expect some sort of qualitative evaluation showing that the agents in fact behave in a way to minimize surprise.  The agents on their own should show the same qualitative behaviors seen in surprise minimization in single agent settings, but, more importantly, it is important to be clear about the qualitative differences we should expect in multi-agent settings and to verify that those effects occur since there is added complexity with the multi-agent aspects of the problem. 

Without those qualitative effects, empirically the results seem to be marginal, for instance in Table 1 it is difficult to tell which of the results are statistically significant.  It seems like most of the error bars overlap, so the results are statistically insignificant, but only one of the methods is bolded.  To be clear, I do not think that the method needs to surpass all others on success rate, it isn't designed to make those sorts of improvements, but because that is all we have for empirical evaluation the best I can conclude is that the intervention into QMIX does not break the method.

On a related note, I think the work would be much more intuitive if the utility of minimizing surprise in multi-agent RL, and the unique challenges that come with multi-agent RL were made more clear from the beginning.  As it stands, I understand from the introduction I understand that EBM are a natural way to approach surprise minimization in multi-agent RL, but it isn't clear to me why you would want surprise minimization in multi-agent RL particularly, and what is different about multi-agent RL that would effect the sorts of behavior we would expect out of surprise minimization.  I think this could be alleviated by an example showing how minimizing surprise in MARL produces desirable effects, and that doing surprise minimization without considering any multi-agent aspects does not produce that desirable behavior.


Minor point:
* In the final paragraph of page 3, the Q_a is said to be lower bounded by Q^* when it is demonstrated to be upper bounded by Q^*
summary_of_the_review: I will weakly recommend rejection due to the limited qualitative evaluation and otherwise inconclusive empirical results.  I would be willing to reconsider my score if the approach could be show to promote the expected qualitative effects of minimizing surprise in a simple setting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present a method to regularise the learning of Q values within Decentralised partially observable markov decision processes, where the regulariser is one that minimises surprise in some way across the population of agents within the environment.

This it is argued allows the agents to avoid situations in which states are "rapidly changing", instead aiming to reach an equilibrium state where just enough surprise is being experienced as part of a reward maximising objective.

The authors present a series of results in which their method outperforms a number of SoTA alternatives on a reasonable looking set of benchmarks, as well as an ablation study showing the criticality of each proposed component.

main_review: There is, what appears to be, a very interesting argument being made here and it draws together a number of strands of research from different disciplines. The multi-agent domain being a clear candidate where independent gradient ascent from individual agents can result in a myopic trap interfering with convergence to a global objective. Nevertheless, a number of the ideas, methods, equations and results in this paper are not as clear to me as I would like and so I cannot endorse the paper for acceptance.

One of the main themes that was running through the paper was the idea of surprise minimisation and how this has some degree of tension with the surprise maximisation aspect of maximum entropy (which as the authors say is a strategy for ensuring good exploration). However, this tension is never properly resolved to my satisfaction. 

It is clear that the results show advantages over the other SoTA methods, but I am not sure that the method could be replicated by a reader and so I am unsure whether these experiments could be validated.

## Detailed notes

The term EBM needs defining when first used.

Should really cite the Dec-POMDP literature rather than the POMDP paper.

p2-3: Is  the joint action a cross product over identical action spaces? Or can each agent have different actions available.

If the observation function is a function (determined) then do all agents get the same observation at time t? Or is it probabilistic but the same distribution for all agents? Or does/can it depend on the agent?

The policy and value function seems to depend on the hidden state.

I think you mean that $\pi^\star_\theta = \pi_{\theta^\star}$ (notationall that the best policy. It's clear what was intended but the notation is sloppy.

In equation 1, b is not very clear. Is this the expectation of the sample average over the minibatch? In which case that is not what the equation says. In short, b should enter into the equation somewhere other than just the subscript to the expectation. Also, sampled from memory R is ill-defined.

Figure 1 does not clearly illustrate what it intends to. There is no description of what the grey surface is, nor of the blue points on this surface and the connecting lines. Yes, the little pictures of robots with a red background are associated with a region of lighter grey, and the green associated with darker grey (which we are told is associated with energy minima), but in all this doesn't leave the reader any the wiser as to what is going on.

One of the arguments to the surprise value function $V_{\text{surp}}$ is $\sigma$ which is described as "deviation...within states for each agent a", but it isn't clear what is meant by this.

The function $V_{\text{surp}}$ is not clear at all. It is said that the $\Tau$ operator is a log-sum-exp contraction for this function but it isn't clear how to calculate this in the first place. Maybe I am missing something.
summary_of_the_review: I confess that I miss the point of the paper. I am not confident that I could recreate the method and this potentially of by-product of key bits of information being excluded from explanations. I am fairly confident that the authors have a strong technical understanding and justification for what they have done, but this is not expressed clearly in the paper, at least not in a way that I can decode.

## Update after rebuttal

My concerns have been partly addressed by clarification on what measures of surprise can be used and how these would be estimated. The publication of the code also goes some way to addressing my concerns. The other updates also improve the paper and the explanations and so I am changing my recommendation to weak accept.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the value factorization approach for solving multi-agent problems. It  considered the spurious surprise states in execution and proposed to address this problem through the proposed free-energy minimization framework.

main_review: Strengths:
--The presentation is clear and easy to follow. 
--The empirical results and ablation studies are sufficient.


Weaknesses (Questions):

1- The ``deviation $\sigma$ within states for each agent $a$ ” is not clear to me. As the authors only defined observations $O$ for each agent in Sec 3.1, is  $\sigma$ the deviation within all agents’ observations?


2- in Eq. (7), should the second $Q(u,s;θ) $ read as $\hat{Q}(u,s;θ)$?

3- Eq.(3) is not clear to me. The surprise is also not defined yet. Why should Eq(3) represent a global surprise? How minimizing Eq(7) can minimise surprise? 

4- Table 1 shows that EMIX improves the winning rate in most of the considered tasks. But it seems the improvement is marginal. This is also the case for the results in Figure 3.

5- Only using SC-II micromanagement is not convincing enough. It is better to include results on other different benchmarks.

summary_of_the_review: 
This paper is attempting to extend the surprise minimisation to the multi-agent learning regime. My major concern is that Eq.(4) seems to be solely a marginally new way of representing the Q-function. It is not different from the typical Q-Mix or Q learning. 

The core message of how Eq(4) or Eq(7) can minimise surprise is not clear to me either.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduced a suprise term in optimizing the policies (action-value functions in Q-learning) for solving the non-stationary challenge due to the rapid changes from the environment for MARL scenarios. This work not only proposed the concept of suprising value in the context of MARL, but also gave an operator (i.e. a contration) that can make the sequence of suprising values converges to a fixed point. The general sketch of the proofs is correct. Also, the authors showed that the convex conjugate of the suprising value operator is analogous to the minimization of the uncertainty among agents. By incorporating the suprising value into the Q-learning algorithm, the term led by the ratio between target and behaviour suprising value (suprising ratio) can also be interpreted as an intrinsic reward. Moreover, the authors also compare the proposed surprising minimization objective with the Soft Q-learning objective. 
main_review: ## Strengths

1. The writing of this paper is clear enough and the presentation manner is concise.
2. The proposed method is novel to some extent.
3. The comparison between the proposed method and other related works is good.
4. Ablation studies are conducted to demonstrate the effects of hyperparameters.

## Weaknesses

1. Although the authors claimed that the proposed learning framework is for MARL, the whole theoretical framework is weakly connected to the context of MARL (in decentralized manner). The only connection is the assumption that the sum of suprising values instead of the single global surprising value (that can be seen as a centralized optimization that is equivalent to the single agent case). In my view, it could be more appropriate to write this paper as a single agent RL work and the contributions would not be reduced. For now, this work is actually a theory based on the single agent scenario and an trivial extension to the multi-agent scenario given the assumption of the global suprising value being equal to the sum of decentralized surprising values.
2. As the authors said in Theorem 3, the proposed surprising value objective can be reduced to the soft Q-learning objective. Could the authors explain in more details why the the proposed surprising value objective is necessary? The biggest concern from me is that it still needs to learn the thermal equilibrium of the surprising value. In other words, during the learning process the proposed surprising value is **not** constant, which may conflict with the claims in the paper.
3. In the concergence analysis, the authors subtly divided the suprising terms and the orginal value terms so that these two terms can be proved separately. **Nevertheless**, there exists a potential issue that the time scales of these two convergences (maybe inaccurate) are actually inconsistent. Specifically, the oprator of the orginal values is with respect to the dynamic environment, so it is corresponding to the transition between different states. While the oprator of the surprising value is actually for the static scenario (if I understand correctly), i.e., for a given state-action pair. Therefore, it could lead to a potential issue that the rate $\beta$ for the operator of surprising value may impact the convergence of optimal policies. Could the authors discuss more about this issue? The convergence rate of the whole sequence needs to be discussed in my view, according to the $\beta$.
4. The authors have compared the proposed algorithm with MARL algorithms, but most of them are irrelevant to the surprise term or intrinsic reward (as the authors claimed the relation in the paper). Can the authors show more comparisons with these highly relevant baselines so as to show the effectiveness of the proposed method?
5. About the SMAC experiment, some of the experimental results shown in the paper are far lower than the existing results, e.g. 5m\_vs\_6m, 3s\_vs\_5z, 10m\_vs\_11m. Can the authors give a convinciable reason or rerun these results by the latest baseline implementation from Pymarl.
6. As I concerned in theoretical analysis above, the choice of $\beta$ is important that is demonstrated in the abalation study. Can the authors show some laws or insights on how to select $\beta$?
summary_of_the_review: In summary, this work proposes an interesting problem and give a thourough analysis on the proposed method. However, due to the multiple concerns on the theoretical and experimental results, at the moment I can only recommend reject (but marginally below the threshold).
review_title: null

Meta-review: The paper explores surprise minimization in multi-agent learning by using free energy across all agents in a multi-agent system. A temporal EBM represents an estimate of surprise which is minimized over the joint agent distribution. Empirical studies on the proposed method are conducted. This paper builds in an interesting direction around surprise minimization in multi-agent learning by using the energy-based framework, but the presentation of the method seems to need more efforts to be improved to avoid confusion. 

The discussion between authors and reviewers is summarized below: The major concerns of Reviewer doix include that: (i) the empirical results are not compelling, (ii) qualitative results are missing, and (iii) the motivation of surprise minimization in multi-agent RL is unclear. After the rebuttal, the authors addressed the concerns of Reviewer doix, who changed his/her score from 5 to 6. The major concern of Reviewer zFND comes from the understanding and justification of the paper. After the rebuttal, the concerns of Reviewer zFND  have been partially addressed by clarification on what measures of surprise can be used and how these would be estimated. Reviewer zFND eventually changed his/her score from 5 to 6.  Also, most of the concerns about theory and experiments from Reviewer 8Wiu have been addressed after the rebuttal. Reviewer 8Wiu accordingly changed the rating from 5 to 6. Reviewer g9cM is still not satisfied with the authors' answers, and his/her concerns regarding some technical issues remain and points out that the current paper has many inconsistencies across the writing that make it hard to evaluate the soundness and correctness of the results.  

After the rebuttal, the author successfully addressed most of the concerns from 3 of 4 reviewers, but the overall rating of the paper is on a borderline level. Given the fact that the paper still has some unaddressed concerns from Reviewer g9cM, and other reviewers actually do not champion the paper. The AC tends to recommend rejecting the paper at the current stage. AC urges the authors to improve their paper by including all the suggestions provided by the reviewers, and then resubmit it to a future venue.
Decision: Reject