Surprise Minimizing Multi-Agent Learning with Energy-based Models
Authors: Karush Suri
Xiao Qi Shi
Konstantinos N Plataniotis
Yuri Andrew Lawryshyn
Venue: NeurIPS 2022 Accept (2022)
Keywords: Energy-based Models
Multi-Agent Learning
Surprise Minimization

Abstract: Multi-Agent Reinforcement Learning (MARL) has demonstrated significant suc2 cess by virtue of collaboration across agents. Recent work, on the other hand, introduces surprise which quantifies the degree of change in an agent’s environ4 ment. Surprise-based learning has received significant attention in the case of single-agent entropic settings but remains an open problem for fast-paced dynamics in multi-agent scenarios. A potential alternative to address surprise may be realized through the lens of free-energy minimization. We explore surprise minimization in multi-agent learning by utilizing the free energy across all agents in a multi-agent system. A temporal Energy-Based Model (EBM) represents an estimate of surprise which is minimized over the joint agent distribution. Our formulation of the EBM is theoretically akin to the minimum conjugate entropy objective and highlights suitable convergence towards minimum surprising states. We further validate our theoretical claims in an empirical study of multi-agent tasks demanding collabora14 tion in the presence of fast-paced dynamics. Our implementation and agent videos are available at the Project Webpage.

Surprise Minimizing Multi-Agent Learning with Energy-based Models
Karush Suri', Xiao Qi Shi?, Konstantinos Plataniotis', Yuri Lawryshyn' 1 University of Toronto, RBC Capital Markets karush.. cerrush.uuriman suri@mail utoronto. ca
Abstract
Multi-Agent Reinforcement Learning (MARL) has demonstrated significant suc- cess by virtue of collaboration across agents. Recent work, on the other hand, introduces surprise which quantifies the degree of change in an agent's environ- ment. Surprise-based learning has received significant attention in the case of single-agent entropic settings but remains an open problem for fast-paced dynamics in multi-agent scenarios. A potential alternative to address surprise may be realized through the lens of free-energy minimization. We explore surprise minimization multi-agent learning by utilizing the free energy across all agents in multi-agent system. A temporal Energy-Basee Model (EBM) represents an estimate of surprise which is minimized over the joint agent distribution. Our formulation of the EBM is theoretically akin to the minimum conjugate entropy objective and highlights suitable convergence towards minimum surprising states. We further validate our theoretical claims in an empirical study multi-agent tasks demanding collabora- tion in the presence of fast-paced dynamics. Our implementation and agent videos are available at the Project Webpage.
Introduction
The rise of RL has led to an increasing interest in the study of multi-agent systems [3], commonly known as Multi-Agent Reinforcement Learning (MARL). In the case of partially observable settings, MARL enables the learning of policies with centralised training and decentralised control [26]. This
has proven to be useful for exploiting value-based methods which motivate collaboration across large number of agents. Bul how do agents behave in the presence of sudden environmental changes?
Consider the problem of autonomous driving wherein a driver (agent) autonomously operates vehicle in real-time. The driver learns to optimize the reward function by maintaining constant speed and covering more distance in different traffic conditions. Whenever the vehicle approaches an obstacle, the driver acts to avoid it by utilizing the brake and directional steering commands. However, due to the fast-paced dynamics of the environment, say fast-moving traffic, the agent may abruptly encounter an obstacle (a person running across the street) which may result collision. Irrespective of the optimal action (pushing brakes) executed by the agent, the vehicle may fail to evade the collision as result of the abrupt temporal change.
The above arises consequence of surprise, which is defined statistical measure of uncertainty. Surprise minimization [3] a recent phenomenon observed the case of single-agent RL methods which deals with environments consisting of rapidly changing states. In the case of model-based RL [24], surprise minimization is used as an effective planning tool in the agent's model [3]. In the case of model-free RL, surprise minimization is witnessed as intrinsic motivation [1, 36] or
generalization problem [9]. On the other hand, MARL does not account for surprise across agents as result of which agents remain unaware of drastic changes in the environment [35]. Thus, surprise minimization the case of multi-agent settings requires attention from a critical standpoint.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

A potential pathway to treat surprising states may be realized in light of fnee-energy minimization. The free-energy principle depicts convergence local niches and provides a general recipe for stability among agents. Through this lens, we unify surprise with free-energy in the multi-agent setting. We construct temporal EBM which represents an estimate of surprise agents may face in the environment. All agents jointly minimize this estimate utilizing temporal difference learning upon their value functions and the EBM. Our formulation of free-energy minimization theoretically akin to minimizing the entropy in conjugate gradient space. This insight provides a suitable convergence result towards minimum surprising states (or niches) of the agent state distributions. In an empir- ical study of multi-agen tasks which present significant collaboration bottlenecks and fast-paced dynamics, we validate our theoretical claims and motivate the practical usage of EBMs in MARL.
2 Related Work
Surprise Minimization:
Despite the recent success of value- basled methods [39, 22] RL agents suffer from spurious state spaces and encounter sudden changes in trajectories. Quantitatively, surprise has been studied as a measure of deviation [3, 9] among states encountered by the agent during its interaction with the environment. While exploring [7, 56] the environment, agents tend to have higher deviation among states which is gradually reduced by gaining a significant understanding of state-action transitions. In the case of and plan effectively for future steps. On the other hand, in the case of model-free RL, surprise results in sample-inefficient learning [1]. This is primarily addressed by making use of rigorous exploration strategies [52, 31.. High-dimensional exploration further requires extrinsic feature engineering [27] and meta models [16]. A suitable way to tackle high-dimensional dynamics is by utilizing surprise as a penalty on the reward [9]. This leads to improved generalization for single-agent interactions [45]. Our proposed approach is parallel to the aforesaid methods.
RL,, agents can leverage spurious experiences [3]
Energy-based Models: EBMs have been successfully implemented in single-agent RL methods [42, 19]. These typically make use of Boltzmann distributions to approximate policies [32]. Such a formulation results in the minimization of free energy within the agent. While policy approximation depicts promise in the case of unknown dynamics, inference methods [57] play a key role in optimizing goal-oriented behavior.
A second type of usage of EBMs follows the maximization of entropy [65]. The maximum entropy framework [20] highlighted in Soft Q-Learning (SQL) [19] allows the agent obey a policy which maximizes its reward and entropy concurrently. Maximization of agent's entropy results in diverse and adaptive behaviors [64] which may be difficult accomplish using standard exploration techniques [7, 56]. The maximum entropy framework is akin to approximate inference in the case of policy gradient methods [49]. Such a connection between likelihood ratio gradient techniques and energy- based formulations leads to diverse and robust policies [17]. Furthermore, their hierarchical extensions [18] preserve the lower levels of hierarchies. In the case of MARL, EBMs have witnessed limited applicability as result of the increasing number of agents and complexity within each agent [8]. While the framework is readily transferable to opponent-aware multi-agent systems [63], cooperative settings consisting of coordination between agents require firm formulation of energy. This formulation must be scalable in the number of agents [15] and account for environments consisting of spurious states [62]. Our theoretical formulation is motivated by these methods in literature.
3 Preliminaries
Multi-Agent Learning: We review the cooperative MARL setup. The problem is modeled as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) [43] defined by the tuple (S, A, r, N, P. 2, O,,) where the state space S and action space A are discrete, r : S x AA + [mmnns maz]] presents the reward observed by agents E N where N is the set of all agents, P: SSSSxx x A + J0. presents the unknown transition model consisting of the transition probability to the next state s' S given the current state s E S and joint action u E A (a combination of each agent's 's action u(a E AA") at time step t and y is the discount factor. Our setting consists of the finite-horizon discounted problem case where episodes terminate at timestep T with the terminal state being 8T. As a result, task returns remain bounded for each episode. We consider a partially observable setting in which each agent n draws individual observations z E according to the
observation function (s,, u) S x A + Z. We consider a joint policy To (uls) as a function of
2

---

model parameters 8. Standard RL defines the agent's objective to maximize the expected discounted reward Exx, 12 1 Uz)] function of the parameters 0. The joint action-value function for agents is represented as Q(u, 8; 0) = TT EEi= 7'r(s, w)|s St, u uz] which the expected sum of payoff's obtained in state upon performing action u by following the policy иT. We denote the optimal policy Te- shorthand π²) such that Quu, s; 8*) 2 Quu, s; 0yvs E S, u E A. In the case of multiple agents, the joint optimal policy can be expressed as the Nash Equilibrium [40] of the Stochastic Markov Game as π = (Tri* .2.- ....TT.,, such that Q(u", s; 0*) 2 Q(u", fy E S,u u E A, a E N. Q-Learning is off-policy, model-free algorithm suitable for continuous and episodic tasks. The algorithm uses semi-gras amii-gradient descent to minimize the Temporal Difference (TD) error in Equation 1.
{1+
mmax (uu':(((')s)))>) -((")))Q(uu,))))) Q(u,s:0)) E'EA
L(0) = E
(1)
where y = r mmax s';00) 0-) is the TD target consisting of 0 as the target parameters and R denotes the replay buffer.
EBMs [29, 30] have been successfully applied in the field of machine learn- ing [55] and probabilistic inference [37]. A typical EBM & formulates the equilibrium probabilities [47] P(u, h)
Energy-based Models:
exp (-p(_E(),A) via Boltzmann distribution [32] where and are the values
Σa,h (-E(,kk)
of the visible and hidden variables and é and h are all the possible configurations of the visible and hidden variables respectively. The probability distribution over all the visible variables can be obtained by summing over all possible configurations of the hidden variables. This is mathematically expressed in Equation 2.
En exp (()())))) P(v)
Σs, exp (-E(,
Here, E(v, ht) is called the equilibrium free energy which the minimum of the variational free energy and Es,h exp (6, is the partition function.
Energy-based Surprise Minimization
We begin by constructing surprise minimization as an energy-based problem the temporal setting. The motivation behind an energy-based formulation stems from rapidly changing states unde- sired niche among agents the case of niche, we further construct method which incorporates the theoretical aspect of the study.
settings. To steer agents away from this
4.1 The Surprise Minimization Objective
To make analysis tractable towards valid function spaces and surprising states, we take into account two assumptions which form the central basis of surprise minimization among multiple agents.
Assumption 1. (Completeness of value function space) The space II S A of all Q value functions (ss,u) u) E II, Vs E S, Vu E A a nonempty complete metric space.
Assumption restricts the formulation of individual agent value functions Qa to the nonempty complete metric space. nonempty space confirms the presence candidate functions Qo upper bounded by the optimal function Q" i.e.- Q. < Q",, Va E N [5]. The completeness counterpart, on other hand, provisions fixed interior int IT for optimization [6].
Assumption 2. (Constant surprise at Equilibrium) In the limit of convergence lim to an x.-x*
optimal policy n², all agents E N incur a finite surprise Ç > between consecutive states and s' until termination state sT.
3

---

Assumption 2 is directly based on the constant and continuous temporal aspect of surprise minimiza- tion [50, 12]. Corresponding to the lifetime of each agent E .,, desired minima bakes in the optimal distribution of actions which correspond to minimum but finite instantaneous surprise.
We formulate the energy- objective consisting of surprise as function of states s. joint actions and standard deviation o of observations for each agent In the case of high-dimensional state spaces (such multiple opponents), informs agents of the abrupt statistical change that would take place upon executing action u. We formulate surprise as V surp(s o)) which serves as an uncertainty quantifier Unc(s,a) of the state-action distribution. Here V (8, u, σ)) denotes the surprise value function which serves as mapping from agent and environment dynamics to surprise. Define an operator presented in Equation which sums surprising configurations across all agents.
TV (s, 0) log Σ cxp (Vsurp(s, u, ())
(3)
Remark 1. TV surp (8, u, o) intuitively provides global estimate of surprise. If all agents are equally likely to face surprising state, then TV Susppp (s, u, d)) captures their individual contributions.
The formulation makes of the soft-maximum operator [2]. The operator Surupp(s u. or) is similar to prior energy formulations [19] where the energy across different actions is evaluated. In our case, inference is carried out across all agents with actions as prior variables. However, in the special case of using an EBM as a Q-function, our approach suitably generalizes to the above methods (details in Appendix B).
Our choice of Vaupp u, 0) is based on its unique mathematical properties which result in better convergence. Of these properties, the most useful result is that T forms a contraction on the surprise value function "surp(s: u, o)) indicating a guaranteed minimization of surprise within agents. This is formally stated in Theorem I while utilizing the completeness criterion of Assumption I which provides a tractable value function space. All proof' are deferred to Appendix A.
Theorem 1. Given surprise value function Ve Viurp (s, u, o) Va E N, the energy operator TVa surp (s, O) = log Ea exp VVupp(s o)) forms contraction on Va Vaurp (8, O)
Theorem provides a suitable guarantee of TV ( u, o)) converging to a fixed point niche. The contraction result is directly based on Banach's fixed point property and suggests the generalization of convergence in any nonempty complete metric space (X,d) d) [5].
We now consider a weighted combination of Q(s, u) with TV surp (s, u, σ) wherein we denote 3 as temperature parameter,
su, = Q(u, s; log exp (Va (V,urpss,u u))))
(4)
Remark 2. Equation is an instance of value function regularization wherein the Q values are subject to joint penalty while observing surprising states.
Interestingly, upon considering the Legendre transform (x) [6, 14] (convex conjugate function corresponding to the conjugate space X' of a differentiable function fz)) of TVa surp (8, U, a), we
obtain the following.
((xx= = sup
(x-z- (=-z-((z)) f(z) = =TV VVump (s)
(5)
#Edom f
(x)x== = Σ x: xlog(a)) Va[(a)
66)
Remark The Legendre Transform of T as value function regularization Q Q - (x) corresponds to the minimum entropy formulation in conjugate space Ex,, (r(ss, ue) xH(x)) for Vzf(z) E E
0) given by f (x) = x log(x) when utilized
Based on the above insight, minimizing entropy to express v_f(2) in conjugate space is akin to minimizing uncertainty among agents in the value function space IT. Intuitively, H(x:) denotes the uncertainty for each agent E N in the multi-agent population which is directly related to its

---

ability of accurately interpreting the environment. Minimizing H(x)) leads to an increase in the expressiveness of value function. This in turn, induces an expressive state visitation distribution which steers the agent away from sudden changes in its environment. Note that the setting does not minimize entropy in value function space which would stand contrary to the maximum entropy formulation [20] (see Appendix B).
Figure 1 presents an illustration of the intuition behind sur- prise minimization using the energy-based scheme. Agents collaborate in partially-observe Interpreting the space of all surprising states as an energy landscape, MARL agents move from high energy states to low energy states which consist of minimum surprise. During training, agents train to find policies which not only provide rewarding actions, but also avoid risky states by minimizing Surpp (s, u, r). Seeking these states leads to finding the min- ima the energy landscape. Thus, it is by virtue of regularized value estimates Q that the minimization scheme informs agents of joint surprise.
worlds attain a joint niche.
Agent populations (robots) traverse the energy landscape (in grey) during update steps (blue) to seek energy minima (darker shade at center). This results surprise minimization from high (red) to low energy (green) niches.
Figure 1:
4.2 Surprise
Minimization with Function Approximation
We utilize the above insights surprise-based regularization in the TD learning setting. Upon replacing Quu, 8; 0) with Qu,, s, 0) in the RL construction of Equation one obtains the following,
N
2
L(O)
E s,
(-
s; log Σ
exp (Vurp u, o)))
2
where ý is given by the following expression,
i = r ymax + Bog
exp (Vaurp (s', (s u2, o'))
(7)
Collecting the log terms yields the following.
1 (r 7max
L(0)
E s
2
27
Σa exp (Vurp((e) (s', ('''
+/3 log
QQu s;
Ee=1 exp (Vsurp(e (s, u,)))
((0) E s.u.d (2 "i=tma(uudd BE Q(u, s; + 27
Here, E is defined as the surprise ratio. The surprise value function Viuep(s' (s', (s',,,,, u', o') is expressed as the negative free energy and DMY exp (Varp.(s u, a)) as the partition function of a conventional EBM described in Equation 2. Alternatively, Vsurp(s; (8, u, or) can formulated as the negative free
energy with Σn' exp (Verr (s', , u', .a)) as the partition function. The TD objective incorporates the minimization of surprise across all agents as minimizing the energy in rapidly changing states.
Remark 4. The above formulation of BE can be realized as intrinsic motivation steering the agent towards subgoals with reduced surprise.
The energy formulation E provides a tractable distribution over all surprising configurations in the state space S. This guarantees convergence to minimum surprise at optimal policy n² and is formally expressed in Theorem 2 (see Appendix C for detailed convergence analysis).
5

---

Theorem 2. Upon agent's convergence optimal policy x², total energy of n², expressed by E' will reach a thermal equilibrium consisting of minimum surprise among consecutive states and s'.
Theorem demonstrates an intuitive convergence result of agent populations collaborating to reside in mutual ecological niche [12]. The multi-agent population with minimum surprise exhibits the optimal policy π¹ which results in minimum energy corresponding to each surprising state in the state distribution S. Orthogonally, agents may continue experience finite and constant surprise in the long-hone long-horizon while acting optimally to visit non-surprising and rewarding states. This presents surprise minimization as secondary surrogate objective MARL.
4.3 Energy-basee MIXer (EMIX)
Global siaaee encuder
Surpriss enezteer
o
Kaup (s.
0,
0
deviation encoder
Valuo
log
Encoders
Figure The EMIX architecture for learning surprise across global states.
Based theoretical analysis, incorporate learning of surprise global intrinsic motivation across all agents in the multi-agent system. A global estimate of surprise, following the energy operator TVa sump(e (s, u, o), is befitting from a computational perspective estimate of surprise for each agent may be intractable to obtain due to the non-stationarity of the environment. Instead, we seek minimize surprise jointly across all agents using an expressive Energy-based MIXer (EMIX) architecture which is compatible with any multi-agent RL algorithm. Figure illustrates our learning scheme.
well. An individual
Learning of surprise in the high- high-dimensional value function space is cumbersome with the number of actions scaling linearly in the number of agents. This imposes inherent restriction learn global surprise efficaciously across all agents at a given timestep. Towards this goal, EMIX encodes individual value functions QI, Q2, ... Qn corresponding each agent using local value encoders. These encoders capture the local change in value functions arising over subsequent TD learning iterations [60]. A global state encoder maps environment states 81, $2, ... 8T low dimensional representation. Further, a state deviation encoder encodes deviations across all states 81: 82,
within the given batch. Akin to model-based method [23], the state deviation encoder accounts for uncertainty in an agent's 's state visitation distribution. Note that the encoder does not construct an explicit model of states, but only represents their variation in the agent's environment. This insight essential to account for abrupt dynamics encountered by agents. Representations obtained from state and value function encoders are concatenated and compressed using final surprise encoder which estimates a distribution of surprise values. The distribution implicitly represents the density of states wherein an agent may encounter most surprise. A value estimate Varp(s, u, or) sampled from surprise distribution depicts the variational free energy configuration upon application of T which serves as global intrinsic motivation. Practical training of EMIX proceeds with backpropagation [46] using gradient descent and the reparameterization trick [25] for sampling of Va Vaup (s, u, o).

---

4.4 Practical Implementation
Algorithm presents the EMIX framework (in green) combined with QMIX [44], an off-the-shelff MARL algorithm. The total Q-value Q is computed by the mixer network with its inputs as the Q-values of all the agents conditioned on s via the hypernetwrrs.. Similarly, the target mixers approximate O conditioned on 8', In order to evaluate surprise within agents, we compute the standard deviations and o' across all observations and z' for each agent using and s' respectively, The surprise value function, called the Surprise Mixer, estimates surprise Veurp (s, u, σ)) conditioned on 8, u and σ. The same computation repeated using the Target-Surprise SurrttSuprrie---Miii Mixer for estimating surprise V (s', , u', o') within next-states in the batch. Application of the energy operator along the non-singleton agent dimension for V (s, u, o)) and V surp (s', u', o²) yields the energy ratio E which is used in Equation 8 to evaluate L(0). We then use batch gradient descent to update parameters of the mixer Target parameters 0,7 are updated every update interval steps.
Algorithm 1 Energy- MIXer (EMIX)
1: Initialize Φ, 0, 0T ... O agent and hypernetwork parameters.
2: Initialize learning rate a, temperature 3 and replay buffer R.
3: for environment step do
4: u (W1, U2.... uN)
5: R + R [ u,,,
6: if [R] batch-size then
7:
for random batch do
8:
Qt <- Mixeer-Nerworr(), Q2..,,QN...
9:
O < -Trrget-Mixers(1, Q2..., Q &') Vi 1, 2..,
Calculate and a' using and
11:
Va (s, (s,u,)) u. f- Surprise- Va
a)
12:
surp (5', ut, od)++ + Target-Mixer ,,',,,,,,,,,,,
22N exp (Vanep (o' (o/,ul,o (o')
13:
E log
EA= exp (Vitep (s.u,)))
14:
Calculate L(0) using E in Equation 8
8 a (()
16:
end for
end if
18:
if update-interval steps have passed then
0,- 5,+0, vi 1,2.., m
20:
end if
end for
5 Experiments
Our experiments aim to evaluate the theoretical claims presented by EMIX along with its performance to prior MARL methods. Specifically, we aim to answer the following questions;
(1) How does the provision of an EBM for surprise minimization compare to current MARL methods?
(2) Does the algorithm validate the theoretical claims corresponding to its components?
5.1 Energy-based Surprise Minimization
We assess the validity of EMIX, when combined with QMIX, multi-agent StarCraft II microman- agement scenarios [48] as these consist of a larger number of agents with different action spaces. This in turn motivates greater deal of coordination. Additionally, micromanagement scenarios in StarCraft II consist of multiple opponents which introduce greater degree of surprise within consecutive states.
compare our method to prior methods namely; (1) QMIX [44], constituting of nonlinear value function factorization with monotonicity constraints; (2) Value Decomposition Networks (VDN)

---

Scenarios
EMIX
SMiRL-QMIX
QMIX
VDN
COMA
IQI.
3m
94.90=0.39
93.94+0.22
93.43=0022
94.58+0.58
84.75+77.93
94.79+0.50
38_vs_ 4z
97.22-0.73
0.24+0.11
96.01+3.93
94.29+2.13
0.00++0000
59.75-12.22
8m vs. 9m
71.03=2.69
1..90:11.94
68.2812.30
58.8114.68
4.17:10.58
28.48-22.38
10m vs 11m
75.35=2.30
77.85++2.02
70.36:22.87
71811
4.55:10.73
32.27=25.68
so. many baneling
95187=0.16
93.61+0.94
93.35-10.78
92.26:11.06
91665422.6
74.97+6.52
5m__ vs vs_6m
37,07=2.442
33.27+2..9
34,42+42.63
35.63++3.32
0.52++0.13
14.78+22.72
Table 1: Comparison success rate pereentages between EMIX and prior MARL methods on
StarCraft II micromanagement scenarios. EMIX is comparable to improves over QMIX agent. In comparison SMiRL-QMIX, EMIX demonstrates improved minimization of surprise. Results are averaged over 5 random seeds.
[53], consisting linear additive factorization of Q function; (3) Counterfactual Multi-Agent Policy Gradients (COMA) [11], which consist of counterfactual actor-critic updates in centralized critic; and (4) Independent Q Learning (IQL) [54], wherein each agent acts independent of other agents. (5) In order to compare our surprise minimization scheme against pre-existing mechanisms, we compare EMIX additionally to model-free implementation of SMiRL [3] QMIX. We use the generalized version of SMiRL as it demonstrates reduced variance across batches [9]. This implementation is
denoted as SMiRL-QMIX for comparisons. Details related to the implementation of EMIX are
presented in Appendix D.
Table presents the comparison of success rate percentages between EMIX and prior MARL algorithms on 6 StarCraft II micromanagement scenarios. Corresponding to each scenario, algorithms demonstrating higher success rate values in comparison other methods have their entries highlighted in bold (see Appendix E. for statistical analysis). Out of the 6 scenarios considered, EMIX presents higher success rates on of these scenarios depicting the suitability of the proposed approach. In cases of so many baneling and 5m_vs _6m having large number of opponents and greater level of surprise, EMIX aptly improves over prior methods. When compared to QMIX, EMIX depicts improved success rates on all of the 6 scenarios. On comparing EMIX with SMiRL-QMIX, EMIX demonstrates higher average success rates indicating surprise robust policies.
5.2 Ablation Study
We now present the ablation study for the various components of EMIX. Our experiments aim to determine the effectiveness of the energy-based surprise minimization method. Additionally, we also aim to evaluate the utility of dual approximators for surprise estimation in accordance with the precept from RL literature [21, 10].
EMIX Objective: To weigh the effectiveness of energy-based scheme, we ablate the energy operator T and only utilize Vsurp. Since this implementation employs dual approximators V i {1,2} for stability, we call this implementation as TwinQMIX. Thus, we compare between QMIX, TwinQMIX and EMIX to assess the contributions of each of the pro- posed methods.
many beneing
QME EvIx
Figure 3: Ablations for each of EMIX'S component. When compared to QMIX, EMIX and TWinQMIX depict improve- ments in performance and sample effi- ciency.
Figure 3 presents the comparison of average success rates for QMIX, TwinQMIX and EMIX on different scenarios. In comparison to QMIX, TwinQMIX adds stability the original objective by incorporating surprising estimates. On comparing TwinQMIX EMIX we note that dual approximators play little role in improving convergence. Thus, the energy- based surprise minimization scheme is the main facet for significant performance improvement. This is demonstrated in the 5m_vs 6m scenario wherein the EMIX implementation improves the performance of TWInQMIX comparison to QMIX by utilizing a surprise-robust policy. In the case of so_many Juneling scenario which consists of large number of opponents (27 banelings). EMIX tackles surprise effectively by preventing significant drop performance which is observed in cases of QMIX and TWinQMIX.
8

---

Figure 4: Variation of surprise minimization with temperature 3. Learning of surprise is achieved by
making use of suitable value of temperature parameter (3 0.00) which controls the stability in surprise minimization by utilizing E as intrinsic motivation.
Surprise Minimization with Temperature: The im-
portance of / can be validated by assessing its usage in surprise minimization. We observe the variation of E' as it is a collection of surprise-based sample esti- mates across the batch. Additionally,
consists of prior samples V surp (s, O) for V surp(ss (5', ut, o') which makes inference tractable.
0.00
Figure Variation in success rates with temperature B. A value of ß = 0.01 is found to work best.
Figure presents the variation of Energy ratio with
the temperature parameter 3 during learning. We com- pare two stable variations of E at 8 = 0.001 and / = 0.01. The objective minimizes F over the course of learning and attains thermal equilibrium with minimum energy. Intuitively, equilibrium corresponds to convergence to optimal policy ग which validates the claim in Theorem 2. With 0.01, EMIX presents improved convergence and surprise minimization for considered scenarios, hence validating the suitable choice of B. The choice of B is further validated in Figure 5 wherein
of the
0.01 provides consistent stable improvements over other values. Lower values of B, such as B = 0.001, do little to minimize surprise or improve performance.
6 Qualitative Analysis
Figure 6: Task- ha.manybbanllin (left) Behaviors learned by EMIX agents, (right) Behaviors
learned by QMIX agents
Figure 7: Task- 2s_vs_ vs Isc, (left) Behaviors learned by EMIX agents, (right) Behaviors learned by QMIX agents
We visualize and compare behaviors learned by surprise minimizing agents to the prior method of QMIX. Fig. 6 presents the comparison of EMIX and QMIX agent trajectories (in yellow arrows) on

---

the challenging so_ many_ baneling task. The task consists of 27 baneling opponents which rapidly attack the agent team on bridge. QMIX agents naively move to the central alley of the bridge and start attacking enemies early on. While QMIX agents naively maximize returns, EMIX agents learn a different strategy. EMIX agents rearrange themselves first at the corners of the bridge. Note that these corners provide cover from enemy's ss fire. Thus, EMIX agents learn to take cover before approaching the enemy head-on. This indicates that the surprise-robust policy is aware of the incoming fast-paced assault.
As another example, Fig. 7 presents behaviors on the 2s 2s_vs_ __vs__se task wherein two agents must collaborate together to defeat SpineCrawler enemy. The enemy. having long tentacle pointing to the front, chooses attack any one of the agents randomly in front of Additionally, the tentacle has fixed length and cannot extend beyond this range. Random intermittent attacks indicate that the agents face a greater degree of surprise with no prior knowledge of the enemy's 's movement. We observe that QMIX agents take turns to attack the enemy by moving back and forth to minimize damage. EMIX agents, on the other hand, learn different strategy. One of the EMIX agents stands at distance to attack th enemy while the other agent goes around to attack from behind. This indicates that the policy is aware of enemy's 's limited movement.
6.1 Predator-Prey Benchmark
We extend our comparison of EMIX on the Predator-Pres Prey (particle world) tasks. In addition to the difficulty of task, we vary the number of opponents. This helps quantify the variation in performance against increasing level of surprise under fixed dynamics. Table presents average returns. While all agents present comparable performance on the easier tasks, EMIX improves over QMIX and TwinQMIX on the more challenging punish and hard tasks. In the case of punish, EMIX is the only method to achieve greater than 20 returns. Additional results can be found in Appendix E.3.
Scenarios
EMIX
TwirQMIX
SMIRL-QMIX
QMIX
VDN
COMA
IQL
prodator_pey__co caas
40.00 =: 0.13
40.00
40.00 = 0.98
40.00 0.22
38.74
27.49 = 4.26
34.75
2.92
predstor__p ppryoo
40,00
0.72
40.00 1,92
40.00 : 0.27
40,00
0.16
36.23 + 3.19
25.13 2 0.92
31.59
i 0.74
prodator_ prcy_pycyyccpc punish
24.17 3.29
20.32
19.31 + 1.12
14.33 + 3.81
17.21 + 2.31
10.92
+ 4.35
7,86 1.86=3.0
predator prey hard
12.34
3.11 3.11
10.19+
10.47 + 0.83
8.76 + 4.33
5.19 3.97
-437 + 1.53
-9.26 4.844
Table 2: Comparison of average returns between EMIX, its ablations and prior MARL methods on Predator-Prey tasks. EMIX improves over QMIX and SMiRL-QMIX.
7 Discussion
Conclusion: In this paper. we presented an energy- based perspective towards surprise minimization in multi-agent RL. Towards this goal we introduce EMIX, an energy- based intrinsic motivation framework for surprise minimization in MARL algorithms. EMIX utilizes a temporal EBM to estimate and minimize surprise jointly across all agents. Our theoretical claims on the formulation of minimization of temporal energy with surprise are corroborated upon utilizing EMIX on a suite of challenging MARL tasks requiring significant collaboration under fast-paced dynamics.
Future Work: While EMIX serves a practical example of EBMs in cooperative MARL, presents several new avenues for future work. We shed light on 2 such aspects,
(1) Provision of energy-based model naturally raises the question of how can we efficiently sample from the surprise distribution? Advances in sampling methods depict promise towards this aspect.
(2) Although suitable for lower dimensions, the scalability of EBMs towards high dimensional action spaces remains an open question. We conjecture that the utility of density- based methods and generative models can address the scalability gap. These directions are left for future work.
10

---

References
[1] J. Achiam and S. Sastry. Surprise-based intrinsic motivation for deep reinforcement learning,
2017.
[2] K. Asadi and M. L. Littman. An alternative softmax operator for reinforcement learning. In International Conference on Machine Learning, 2017.
[3] G. Berseth, D. Geng, C. Devin, D. Jayaraman, C. Finn, and S. Levine. Smirl: Surprise minimizing rl in entropic environments. 2019.
[4] D. P. Bertsekas. Abstract dynamic programming. Athena Scientific Nashua, NH, USA, 2018.
[5] D. P. Bertsekas and J. N. Tsitsiklis. Neuro-dynamic Programming, volume 1. Athena Scientific,
1995.
[6] Boyd and Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
[7] Y. Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros. Large-scale study of curiosity-driven learning. ICLR, 2019.
[8] L. Buçoniu, R. Babuška, and B. De Schutter. Multi-agent reinforcement learning: An overview. In Innovations in agent systems and applications- 2010.
[9] J. Z. Chen. Reinforcement learning generalization with surprise minimization, 2020.
[10] L.. Chenghao, T. Wang, C. Wu, Q. Zhao, J. Yang, and C. Zhang. Celebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
[11] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients, 2017.
[12] K. Friston. The free-energy principle: unified brain theory? Nature reviews neuroscience, 11(2):127-138, 2010.
[13] S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods, 2018.
[14] B. Gao and L. Pavel. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint arXiv: 00805, 2017.
[15] J. Grau-Moya, F. Leibfried, and H. Bou-Ammar Balancing two-player stochastic games with soft q-learning. arXiv preprint arXiv:/800 032 16, 2018.
[16] A. Gupta, R. Mendonca, Y. Liu, P. Abbeel, and S. Levine, Meta-reinforcement learning of structured exploration strategies. In Advances in Neural Information Processing Systems 31. 2018.
[17] Haarnoja. Acquiring Diverse Robot Skills via Maximum Entropy Deep Reinforcement Learning. PhD thesis, UC Berkeley, 2018.
[18] Haarnoja, K. Hartikainen, P. Abbeel, and Levine. Latent space policies for hierarchical reinforcement learning. arXiv preprint arXiv: (tt4:v880.000000 2018.
[19] Haarnoja, H. Tang, Abbeel, and Levine. Reinforcement learning with deep energy-based policies. arXiv preprint /702.087 2017.
[20] Haarnoja, Zhou, P. Abbeel, and Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement leamning with stochastic actor. arXiv preprint arXiv: arXiv:/801. 2018.
[21] H. V. Hasselt, A. Guez, and Silver. Deep reinforcement learning with double g-lennning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 2016.
11

---

[22] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. arXiv preprini arXiv: 171000299, 2017.
[23] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, 2019.
[24] L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, A. Mohiuddin, R. Sepassi, G. Tucker, and H. Michalewski. Model-based reinforcement learning for atari, 2019.
[25] D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. In 2nd International Conference Learning Representations, 2014.
[26] L. Kraemer and B. Banerjee. Multi-agent reinforcement learning as rehearsal for decentralized planning. Neurocomputing, 190, 02 2016.
[27] T. D. Kulkarni, K. Narasimhan, A. Saeedi, and J. Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems, 2016.
[28] A. Kumar. A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. NeurlPS 2020, 2020.
[29] Y. LeCun, Chopra, R. Hadsell, M. Ranzato, and Huang. A tutorial on energy-based learning. Predicting structured data, 1, 2006.
[30] Y. LeCun, S. Chopra, M. Ranzato, and F.-J Huang. Energy-based models in document recognition and computer vision. In Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), volume 1, 2007.
[31] L. Lee, B. Eysenbach, E. Parisotto, E. Xing, S. Levine, and R. Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint 1rXiv::906.055 2019.
[32] S. Levine and P. Abbeel. Learning neural network policies with guided policy search under unknown dynamies. In Advances in Neural Information Processing Systems, 2014.
[33] M. A. Lones. How to avoid machine learning pitfalls: a guide for academic researchers. arXiv preprint arXiv:2108 024977 2021.
[34] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch. Multi-agent actor-critic for mixed cooperative-comppttiiee environments, 2017.
[35] L. Macedo and A. Cardoso. The role of surprise, curiosity and hunger on exploration of unknown environments populated with entities. In 2005 portuguese conference on artificial intelligence, 2005.
[36] L. Macedo, R. Reisezein, and A. Cardoso. Modeling forms of surprise in artificial agents: empirical and theoretical study of surprise functions. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 26, 2004.
[37] D. J. C. MacKay, Information Theory, Inference & Learning Algorithms. Cambridge University Press, 2002.
[38] H. B. Mann and D. R. Whitney. On a test of whether one of two random variables is stochastically larger than the other. Annals of Mathematical Statistics, 18, 1947.
[39] V. Mnih, P. Badia, M. Mirza, A. Graves, T. Lillicrap. T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, 2016
[40] Nash. Equilibrium points in n-person games. Proceedings of the National Academy of Sciences, 36(1), 1950.
[41] J. Nocedal and S. Wright. Numerical optimization. Springer Science & Business Media, 2006.
12

---

[42] B. D'Donoghue. R. Munos, K. Kavukcuoglu, and V. Mnih. Combining policy gradient and q q-learning. arXiv preprint arXiv:1611.01626, 2016.
[43] F. A. Oliehoek and C. Amato. A concise introduction to decentralized POMDPs. Springer, 2016.
[44] T. Rashid, M. Samvelyan, C. S. de Witt, G. Farquhar, J. Foerster, and S. Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In ICML 2018: Proceedings of the Thirty-Fifhh International Conference on Machine Learning, 2018.
[45] W. Ren, R. W. Beard, and E. M. Atkins. A survey of consensus problems in multi-agent coordination. In Proceedings of the 2005, American Control Conference, 2005., 2005.
[46] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning Representations by Back- propagating Errors. Nature, 323:533-536, 1986.
[47] B. Sallans and G. E. Hinton. Reinforcement learning with factored states and actions. Journal of Machine Learning Research, 5, 2004.
[48] M. Samvelyan, T. Rashid, C. S. de Witt, G. Farquhar, N. Nardelli, T. J. Rudner, C.-M. Hung, H. S. Torr, J. Foerster, and S. Whiteson. The starcraft multi-agent challenge, 2019.
[49] Schulman, X. Chen, and P. Abbeel. Equivalence between policy gradients and soft q-learning. arXiv preprint arXiv:1704.066 2017.
[50] Schwartenbeck, T. FitzGerald, R. Dolan, and K. Friston. Exploration, novelty, surprise, and free energy minimization. Frontiers in psychology, 4:710, 2013.
[51] K. Son, D. Kim, W. Kang, D. E. Hostallero, and Y. Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, 2019.
[52] B. C. Stadie, S. Levine, and P. Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.00814 2015.
[53] P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, networks for coop-
N. Sonnerat, J. Z. Leibo, K. Tuyls, and T. Graepel. Value-decomposing erative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 18, page 2085-2087, 2018.
[54] M. Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In In Proceedings of the Tenth International Conference on Machine Learning, 1993.
[55] Y. W. Teh, M. Welling, S. Osindero, and G. E. Hinton. Energy-based models for sparse overcomplete representations. Journal of Machine Learning Research, 4, 2003.
[56] S. B. Thrun. Efficient exploration in reinforcement learning. 1992.
[57] M. Toussaint. Robot trajectory optimization using approximate inference. In Proceedings of the
26th annual international conference on machine learning, 2009,
[58] O. Vinyals, I. Babuschkin, W. Czarnecki, M. Mathieu, A. Dudzik, J. Chung. D. Choi, R. Powell, Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, Danihelka, A. Huang, L. Sifre, T. Cai, J. Agapiou, M. Jaderberg, and D. Silver. Grandmaster level in stareraft ii using multi-agent reinforcement learning. Nature, 575, 11 2019.
[59] J. Wang. Z. Ren, T. Liu, Y. Yu, and C. Zhang. Qplex: Duplex dueling multi-agent q-learning. In International Conference on Learning Representations, 2021.
[60] T. Wang. T. Gupta, A. Mahajan, B. Peng. S. Whiteson, and C. Zhang. (RODE): Learning roles to decompose multi-agent tasks. In International Conference on Learning Representations, 2021.
13

---

[61] Y. Wang, B. Han, T. Wang, H. Dong, and C. Zhang. Dop: Off Off-po multi-agent decomposed policy gradients. In International Conference on Learning Representations, 2021.
[62] E. Wei, Wicke, D. Freelan, and Luke. Multiagent soft q-learning. arXiv preprint 18X4 2018.
[63] Y. Wen, Y. Yang, R. Luo, J. Wang, and W. Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning. arXiv preprint arXiv:1901.09207, 2019.
[64] B. D. Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010.
[65] B. Ziebart, A. L. Maas, Bagnell, and A. Dey. Maximum entropy inverse reinforcement learning. In AAAI, 2008.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "zR0dQQk5XLu", "reviewer": "Reviewer_rSgP", "summary": "Surprise, which quantifies the degree of changes in agents' environment, has received significant attention in the case of SARL but remains an open problem for MARL. This paper explores surprise minimization in MARL by utilizing the free energy across all agents. The formulation of the energy-based model is theoretically akin to the minimum conjugate entropy objective. They further validate the theoretical claims in several MARL benchmarks.", "questions": "Some questions are listed above, here are other questions.\n\n1. In Eq. (3), Why is there still a superscript $a$ after summing over $a$?\n\n2. How to learn $V_{surp}^a(s,u,\\sigma)$?  What is the definition and implementation of state deviation $\\sigma$? What does the black and white grid in Figure 2 mean?\n\n3. Why do SC2 and Predator-Prey need surprise minimization?\n\n4. Can the authors illustrate more about \"minimum conjugate entropy objective\"?", "limitations": "The authors have described the limitations of this work. But in the view of reviewer, the main limitation is that some benchmarks are not suitable for surprise minimization. Maybe the reviewer does not well understand surprise minimization, hope the authors can give more intuitive explanations. ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. **Importance of the problem**. The problem this paper considered is important. How agents behave in the presence of sudden environmental changes is critical in both SARL and MARL. And this paper step forwards this topic in MARL.\n2. **The novelty of the proposed method**. The proposed method is novel in a theoretical view. They have deeply investigated the influence of the introduction of energy-based surprise minimization in MARL.\n\n\n\nWeaknesses\n\n1. **Quality of empirical evaluation**. Not adequate. (1) The compared algorithms, QMIX, VDN, COMA, and IQL, are not SOTA anymore. More new baselines are expected, such as RODE, and QPLEX. (2) There are other difficult scenarios of SC2, what are the results of EMIX in these scenarios?\n2. **Significance of results**. Not significant. In Table 1 and Table 2, EMIX only archives incremental improvements over outdated baselines. \n3. **Clarity**. The writing need to be improved. For example, what does Figure 1 want to express? Why is the a superscript $a$ in $V_{surp}^a(s,u,\\sigma)$, as there is no specific information related to agent $a$?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_hT28Z_aWtS", "reviewer": "Reviewer_RyKq", "summary": "This paper proposed incorporating surprise minimization in multi-agent reinforcement learning (MARL) through temporal energy models for capturing the uncertainty across agents. More specifically, this paper formulated an energy function extending the energy-based model theory and proposed an operator to temproally update this energy function. Moreover, this paper analysed the relationship to the soft Q-learning and gave a convergence analysis for the proposed method. With practical function approximation, the proposed algorithm according to the theory called EMIX was shown achieving superior performance with tiny margins compared with the baselines.", "questions": "1. Assumption 2 needs more explanation on why the termination state appears in the discounted problem settings.\n2. I wish the authors give the details on how Eq. (17) is obtained.", "limitations": "The authors gave the limitations and the future work to address it.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The presentation of this paper is good, starting with a reasonable question as the motivation and addresing it by a complete methodology.\n2. The theoretical proofs are generally correct.\n3. The related works discussed in this paper is sufficient.\n4. It compared the relation to maximum entropy framework.\n\n**Weaknesses:**\n1. Assumption 2 is inconsistent with the discounted problem considered in this paper. To my best knowledge, there does not exist a termination state in the context of discounted problems. Nevertheless, it exists in the stochastic shortest path (SSP) problem.\n2. The experimental results of baselines could be underestimated compared with the previous works.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5QlLIrC1-B", "reviewer": "Reviewer_DDcB", "summary": "Paper introduces a new algorithm within the paradigm of Centralised Training and Decentralised Execution for multi-agent games. The authors introduce a new method EMIX - which improves upon QMIX by adding a global surprise minimisation objective to the centralised-value function training. The energy-based model (EBM) approximation is shown to provide stability during training through ablations. \n\n\nThe paper provides strong theoretical justification that 1) the surprise operator converges to a fixed-point, and 2) that as the policy converges to an optimal policy then surprise is also minimised. The paper also demonstrates improved performance than baselines on most games within Starcraft II and through ablations demonstrates the necessity for both EBM approximations and surprise reward shaping.", "questions": "1. Could the authors provide any more indepth analysis of how this outperforms COMA and QMIX, the analysis given in E1 is exceptionally useful and would be appreciated if more could be given of this flavour. I’d even recommend bringing more of this into the main paper.\n\n2. The constant beta is not tuned at all during training - similar to SAC could you not train this?\n\n3. The global state encoder and standard deviation encoders are a slight mystery to me and I’d like more understanding of how they are set up and trained.\n\n4. It is unclear how the V_surp is learnt or obtained. What is the surprise method trained on, how is this estimated during training? \n", "limitations": "The authors have discussed the limitations of this approach as it is scaled. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nStrong theoretical justification\n\nClear Presentation \n\nHigh quality paper\n\nSeems like an original extension in the Centralised Training Decentralised Deployment algorithm family.\n\nWeaknesses:\n\nThe global surprise encoder could be given slightly more description. The diagram is good but an additional example of the flow for interference or training optimisation would be appreciated. \n\nPerformance is only marginally better  (a more challenging benchmark might be necessary). This is the most unconvincing component of the paper but i do appreciate the limited benchmarks available currently.   \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept