Surprise Minimizing Multi-Agent Learning with Energy-based Models
Authors: Karush Suri
Venue: ICLR 2022 Submitted (2022)
Keywords: Multi-Agent Learning
Reinforcement Learning
Energy-based Models.

Abstract: Multi-Agent Reinforcement Learning (MARL) has demonstrated significant success by virtue of collaboration across agents. Recent work, on the other hand, introduces surprise which quantifies the degree of change in an agent's environment. Surprise-based learning has received significant attention in the case of single-agent entropic settings but remains an open problem for fast-paced dynamics in multi-agent scenarios. A potential alternative to address surprise may be realized through the lens of free-energy minimization. We explore surprise minimization in multi-agent learning by utilizing the free energy across all agents in a multi-agent system. A temporal Energy-Based Model (EBM) represents an estimate of surprise which is minimized over the joint agent distribution. Our formulation of the EBM is theoretically akin to the minimum conjugate entropy objective and highlights suitable convergence towards minimum surprising states. We further validate our theoretical claims in an empirical study of multi-agent tasks demanding collaboration in the presence of fast-paced dynamics.

Under review as a conference paper at ICLR 2022
SURPRISE MINIMIZING MULTI-AGENT LEARNING WITH ENERGY-BASED MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Multi-Agent Reinforcement Learning (MARL) has demonstrated significant suc- cess by virtue of collaboration across agents. Recent work, the other hand, introduces surprise which quantifies the degree of change in an agent's 's environ- ment. Surprise-based learning has received significant attention in the case of single-agent entropic settings but remains an open problem for fast-paced dynamics in multi-agent scenarios. A potential alternative to address surprise may be realized through the lens of free-energy minimization. We explore surprise minimization multi-agent learning by utilizing the free energy across all agents in multi- agent system. A temporal Energy-Basee Based Model (EBM) represents an estimate of surprise which is minimized over the joint agent distribution. Our formulation of the EBM is theoretically akin to the minimum conjugate entropy objective and highlights suitable convergence towards minimum surprising states. We further validate our theoretical claims in an empirical study of multi-agent tasks demanding collaboration in the presence of fast-paced dynamics.
sites. google..oooooooooooooooooooo.... com/ wwtp////wwwwwuwwweeeeeeeeee
INTRODUCTION
The rise of RL has led to an increasing interest in the study of multi-agent systems (Lowe et al., 2017; Vinyals al., 2019), commonly known as Multi-Agent Reinforcement Learning (MARL). In the case of partially observable settings, MARL enables the learning of policies with centralised training and decentralised control (Kraemer Banerjee, 2016). This has proven to be useful for exploiting value-based methods which motivate collaboration across large number of agents. But how do agents behave the presence of sudden environmental changes?
Consider the problem of autonomous driving wherein a driver (agent) autonomously operates vehicle in real-time. The driver learns to optimize the reward function by maintaining constant speed and covering more distance in different traffic conditions. Whenever the vehicle approaches an obstacle, the driver acts to avoid it by utilizing the brake and directional steering commands. However, due to the fast-paced dynamics of the environment, say fast-moving traffic, the agent may abruptly encounter an obstacle (a person running across the street) which may result in collision. Irrespective of the optimal action (pushing of brakes) executed by the agent, the vehicle may fail to evade the collision as result of the abrupt temporal change.
The above arises consequence of surprise, which defined statistical measure of uncertainty, Surprise minimization (Berseth et al., 2019) is a recent phenomenon observed in the case of single- agent RL methods which deals with environments consisting of rapidly changing states. In the case of model-based RL (Kaiser et al., 2019), surprise minimization is used as an effective planning tool in the agent's model (Berseth al., 2019) whereas in the case of model-free RL. surprise minimization is witnessed as an intrinsic motivation (Achiam & Sastry, 2017; Macedo et al., 2004) generalization problem (Chen, 2020). On the other hand, MARI does not account for surprise across agents as result of which agents remain unaware of drastic changes in the environment (Macedo & Cardoso, 2005). Thus, surprise minimization in the case of multi-agent settings requires attention from a critical standpoint.
A potential pathway to treat surprising states may be realized in light of free-energy minimization. The free-energy principle depicts convergence local niches and provides general recipe for cognitive stability among agents. Through this lens, we unify surprise with free-energy in the multi-agent setting. We construct temporal EBM which represents estimate of surprise agents

---

Under review as a conference paper at ICLR 2022
may face in the environment. All agents jointly minimize this estimate utilizing temporal difference learning upon their value functions and the EBM. Our formulation of free-energy minimization is theoretically akin to minimizing the entropy in conjugate gradient space. This insight provides a suitable convergence result towards minimum surprising states (or niches) of the agent state distributions. In empirical study of multi-agent tasks which present significant collaboration bottlenecks and fast-paced dynamics, we validate our theoretical claims and motivate the practical usage of EBMs in MARL.
2 RELATED WORK
Despite the recent success of value-based methods (Mnih et al., 2016; Hessel et al., 2017) RL agents suffer from spurious state spaces and encounter sudden changes in trajectories. Quantitatively, surprise has been studied as a measure of deviation (Berseth et al., 2019; Chen, 2020) among states encountered by the agent during its interaction with the environment. While exploring (Burda et al., 2019; Thrun, 1992) the environment, agents tend to have higher deviation among states which gradually reduced by gaining a significant understanding of state-action transitions. In the case of model-based RI,, agents can leverage spurious experiences (Berseth et al., 2019) and plan effectively for future steps. On the other hand, in the case of model-free RL, surprise results in sample-ineffice making use of rigorous exploration strategies (Stadie et al., 2015; Lee et al., 2019). High-dimensional exploration further requires extrinsic feature engineering (Kulkarni et al., 2016) and meta models (Gupta et al., 2018). A suitable way to tackle high-dimensional dynamics is by utilizing surprise as a penalty on the reward (Chen, 2020). This leads to improved generalization for single-agent interactions (Ren et al., 2005). Our proposed approach is orthogonal to the aforesaid methods.
Surprise Minimization:
learning (Achiam & Sastry, 2017). This is primarily addressed by
Energy-based Models: EBMs have been successfully implemented in single-agent RL methods (O Donoghue et al., 2016; Haarnoja et al., 2017). These typically make use of Boltzmann distributions to approximate policies (Levine & Abbeel, 2014). Such a formulation results in the minimization of free energy within the agent. While policy approximation depicts promise in the case of unknown dynamics, inference methods (Toussaint, 2009) play a key role in optimizing goal coienned behavior.
A second type of usage of EBMs follows the maximization of entropy (Ziebart et al., 2008). The max- imum entropy framework (Haarnoja et al., 2018b) highlighted in Soft Q-Learning (SQL) (Haarnoja et al., 2017) allows the agent to obey a policy which maximizes its reward and entropy concurrently. Maximization of agent's ' entropy results in diverse and adaptive behaviors (Zicbart, 2010) which may be difficult to accomplish using standard exploration techniques (Burda et al., 2019; Thrun, 1992). The maximum entropy framework is akin to approximate inference in the case of policy gradient methods (Schulman al., 2017). Such a connection between likelihood ratio gradient techniques and energy-based formulations leads to diverse and robust policies (Haarnoja, 2018) and their hierarchical extensions (Haarnoja al., 2018a) which preserve the lower levels of hierarchies. In the case of MARL,, EBMs have witnessed limited applicability as a result of the increasing number of agents and complexity within each agent (Buşoniu et al., 2010). While the probabilistic framework readily transfcrable to opponent-awe aware multi-agent systems (Wen et al., 2019), cooperative settings consisting of coordination between agents require a firm formulation of energy which is scalable in the number of agents (Grau-Moya et al., 2018) and accounts for environments consisting of spurious states (Wei et al., 2018). Our theoretical formulation is motivated by these methods in literature.
PRELIMINARIES
3.1 MULTI-AGENT LEARNING
We review the cooperative MARL setup. The problem is modeled as Dec-Partially Ob- servable Markov Decision Process (POMDP) (Oliehoek & Amato, 2016) defined by the tuple AS, N, P, Z, 0,7) where the state space S and action space A are discrete, r S x A - ['mnn maaz] presents the reward observed by agents a E N where N is the set of all agents, P:S x Sx8 x A [0, oo) presents the unknown transition model consisting of the transition proba- bility to the next state s' E given the current state s E S and joint action u E A (a combination of each agent's action u² E A") at time step and is the discount factor. We consider a partially observable setting in which each agent n draws individual observations 2 E Z according to the
observation function (s,, u) S x A- Z. We consider a joint policy no(u)s) as a function of
2

---

Under review as a conference paper at ICLR 2022
model parameters 8. Standard RL defines the agent's objective to maximize the expected discounted reward B. Ei ut) function of the parameters 0. The joint action-value function for agents is represented as Quu, 8; 0) = Eng pected of payoffs obtained in state upon performing action by following the policy wy- We denote the optimal policy TT-( shorthand πT) such that Quu, s; 0-)) 2 Q(u, s; 0,ys E S, u E A. In the case of multiple agents, the joint optimal policy can be expressed as the Nash Equi- librium (Nash, 1950) of the Stochastic Markov Game as ' (T , ,....".**) such that Q(u", s; ) 2 Qu" f)vss E S, u A, E N. Q-Learning an off-policy, model-free al- gorithm suitable for continuous and episodic tasks. The algorithm uses semi mrii-gradient descent to minimize the Temporal Difference (TD) error in Equation 1.
7"r(,, u]}s = 8t, = we] which is the ex-
(1, -Q(,,;; 8;0) WEA
L(0)- = E
8,1,8//R
where y mmaxQ 0-) ) is the TD target consisting of 0- the target parameters and R. a'EA
denotes the replay buffer.
3.2 ENERGY-BASED MODELS
EBMs (LeCun et al., 2006; 2007) have been successfully applied in the field of machine learning (Teh et al., 2003) and probabilistic inference (MacKay, 2002). A typical EBM formulates the equilibrium probabilities (Sallans & Hinton, 2004) P(v, h)
exp exp(-E(v.h)) E(v.h)) via Boltzmann [ex (-8(0,A)]] distribution (Levine & Abbeel, 2014) where and are the values of the visible and hidden variables and ê and h are all the possible configurations of the visible and hidden variables respectively. The probability distribution over all the visible variables can be obtained by summing over all possible configurations of the hidden variables. This mathematically expressed in Equation 2.
En (-xp(-E(vr) (-E(v, h))
P(v)
Σa, exp exp
Here, E(v, h)) is called the equilibrium free energy which is the minimum of the variational free energy and Es, exp ((, is the partition function.
4 ENERGY-BASED SURPRISE MINIMIZATION
We begin by constructing surprise minimization as an energy-based problem the temporal setting. The motivation behind an energy-based formulation stems from rapidly changing states an unde- sired niche among agents in the case of partially-obsere settings. To steer agents away from this niche, we further construct method which incorporates the theoretical aspect of the study.
4.1 THE SURPRISE MINIMIZATION OBJECTIVE
To make analysis tractable towards valid function spaces and surprising states, take into account two assumptions which form the central basis surprise minimization among multiple agents.
Assumption 1. (Completeness of value function space) The space II S A of all Q value functions ((s,u) u) E II, Vs E S, Vu E A a nonempty complete metric space.
Assumption restricts the formulation of individual agent value functions Qa to the nonempty complete metric space. A nonempty space confirms the presence candidate functions Qa upper bounded by the optimal function Q*, i.e.- Qo I Q", Va E N (Bertsekas & Tsitsiklis, 1995). The completeness counterpart, on the other hand, provisions a fixed interior int II for optimization (Boyd & Vandenberghe, 2004).
3

---

Under review as a conference paper at ICLR 2022
Assumption 2. (Constant surprise Equilibrium) In the limit of convergence lim to an Eu optimal policy n²,,, all agents a E N incur a finite surprise between consecutive states and s' until termination state 8T.
Assumption 2 is directly based on the constant and continuous temporal aspect of surprise minimiza- tion (Schwartenbeck et al., 2013; Friston, 2010). Corresponding to the lifetime of each agent a E N, desired ecological niche bakes in the optimal distribution of actions which correspond to minimum yet finite instantaneous surprise.
We formulate the energy-based objective consisting of surprise as function of states s, joint actions and standard deviation a of observations for each agent In the case of high-dimenniooal state spaces (such multiple opponents), informs agents of the abrupt statistical change that would take place upon executing action We formulate surprise as TV surp (s, o)) which serves as an uncertainty quantifier Unc(s,a) of the state-action distribution. Here V (s, u, σ)) denotes the surprise value function which serves mapping from agent and environment dynamics to surprise. Define an operator presented in Equation 3 which sums surprising configurations across all agents.
TVsupp u, u) log Σ exp (Vsurp(s, u, (,)
Remark Vurpp( (s, u, о) intuitively provides a d'global estimate of surprise. Ifall agents are equally likely face surprising state, then Surupp u, O) captures their individual contributions.
The formulation makes use of the soft-maximum operator (Asadi & Littman, 2017). The operator TVsu u. o) is similar prior energy formulations (Haarnoja al., 2017) where the energy across different actions is evaluated. In our case, inference is carried out across all agents with actions as prior variables. However, the special case of using an EBM a Q-function, our approach suitable generalizes to the above methods (details in Appendix B).
Our choice of TV Vuurp(s (s, u, is based its unique mathematical properties which result in better convergence. Of these properties, the most useful result is that T forms a contraction on the surprise value function Viurp(5 u, o)) indicating a guaranteed minimization of surprise within agents. This is formally stated in Theorem I while utilizing the completeness criterion of Assumption which provides a tractable value function space. All proofs are deferred to Appendix A.
Theorem 1. Given surprise value function Va Viurp (s, u, Va E N, the energy operator TVa surp (s, o, = log Ea exp (Vaupp (8, r)) forms a contraction on Va Vamp (8, u, o).
Theorem provides a suitable guarantee of ( u, a) converging to a fixed point niche. The contraction result is directly based on Banach's fixed point property and suggests the generalization of convergence in any nonempty complete metric space (X, d) (Bertsekas & Tsitsiklis, 1995).
We now consider weighted combination of Q(s, u) with Viuurp (s, u, wherein we denote 3 as temperature parameter, Qu, s; 0) = Quu, , 0) + log exp (V (8, u, u)))
N
(4)
a=1
Remark 2. Equation an instance of value function regularization wherein the values are subject to joint penalty while observing surprising states.
Interestingly, upon considering the Legendre transform f'xx) (Boyd & Vandenberghe, 2004; Gao Pavel, 2017) (convex conjugate function corresponding to the conjugate space X' of differentiable surp
function fz)) of TV (s, u. a), we obtain the following.
(*(x) = sup (IT - f(x)) f(z) TVe surp (s, o)
(5)
zEdom /
fz(x) = Σ x 2log(x), == ==f(z))
(6)
Remark 3. The Legendre Transform of o Va (su u, o) given by (xx (x)== Ez when utilized as value function regularization Q = Q - f (x) corresponds to the minimum entropy formulation in conjugate space E.,,
XH(x))) for x == Vzf(z) E
4

---

Under review as a conference paper at ICLR 2022
Based on the above insight, minimizing entropy express (z) in con- jugate space is akin to minimizing uncertainty among all agents in the value function space II. Intuitively, (x))) denotes the uncertainty for each agent E N in the multi-agent population which is directly related to its ability of efficaciously interpreting the environment. Minimizing H(x) leads an increase in the expressiveness of value function. This in turn, induces an expressive state visitation distribution which steers the agent away from sudden changes in its environment. Note that the setting does not minimize entropy in value function space which would stand contrary to the maximum entropy formulation Haarnoja et al. (2018b) (see Appendix B).
C-C e-0=
Figure 1: Agent pop- ulations (robots) tra- verse the energy land- scape (in grey) during update steps seek energy minima (darker shade at cen- ter). This results in surprise minimization from high to low energy niches.
Figure presents an intuitive illustration of the objective. The joint agent population aims to minimize surprise corresponding to minimum energy configurations. Agents collaborate eartially-observe worlds to attain joint niche. This local niche implicitly corresponds TVaup(s (s, u, σ)) on the energy landscape. Note that agents act locally with actions conditioned on their own action observation histories. It is by virtue of preconditioned values estimations that the surprise minimization scheme informs agents of joint surprise. Upon population's 's convergence suitable configuration, agents continue to experience minimum (yet finite) surprise arising from evironment dynamics.
to
fixed point of
4.2 SURPRISE MINIMIZATION WITH FUNCTION APPROXIMATION
We utilize the above insights as surprise-basee based regularization in the TD learning setting. Upon replacing (u, s; f) with Q(u, s, ; 6) the RL construction of Equation 1 one obtains the following.
N
27
L(0)
E o R
- (Q(u, s; +)+ + Blog
exp (Vaur((6 (s, u, oi))
2
where 9 = r (m + 3 log En exp surp
o')). Collecting the log terms (s', VVmpp(s'
yields the following,
Σ
Da=1 exp (Ve surp (s', (ss,,'',0 (s',
N
27
r 2
mmax((((ul s';-) 3log
Quu,,, s; s;
S R
Σ exp (Veurp (s, u, 0))
((O) E S.
2 (1+ BE
9; 0) )
Here, E is defined as the surprise ratio. The surprise value function Viurp($ (s', u,, oa) is expressed as the negative free energy and Σ exp (Va (Vurp(ss (s, u, o)) as the partition function of a conventional EBM described in Equation 2. Alternatively, Va (s, a) can be formulated as the negative free energy with EN exp (V (s', u', o')) as the partition function. The TD objective incorporates the minimization of surprise across all agents as minimizing the energy in spurious states.
Remark 4. The above formulation of BE can be realized as intrinsic motivation steering the agent towards subgoals with reduced surprise.
The energy formulation provides a tractable distribution over all surprising configurations in the state space S. This guarantees convergence to minimum surprise at optimal policy n² and is formally expressed in Theorem (see Appendix for a detailed convergence analysis).
Theorem 2. Upon agent's convergence to an optimal policy x", total energy of T", expressed by Ex will reach a thermal equilibrium consisting of minimum surprise among consecutive states and s'.
5

---

Under review as a conference paper at ICLR 2022
Theorem demonstrates intuitive convergence result of agent populations collaborating to reside in a mutual ecological niche (Friston, 2010). The multi-agent population with minimum surprise exhibits the optimal policy x² which results in minimum energy corresponding each surprising state in the state distribution S. Orthogonally, agents may continue to experience finite and constant surprise in the long-horizon while acting optimally to visit non-surprising and rewarding states. This presents surprise minimization as a secondary surrogate objective in MARL.
4.3 ENERGY-BASER MIXER (EMIX)
$2
Global state encoder
Q5
o
amb
Surpitse encoder
Q2
011
Vuur (s, 14,
Qn
0 0 GHD
State deviation encoder
log
Local Value Fnooders
Figure 2: The EMIX architecture for learning surprise across global states.
Based on our theoretical analysis, we incorporate learning of surprise as global intrinsic motivation across all agents in the multi-agent system. A global estimate of surprise, following the energy operator TV Tu (s, u, rr), is befitting from a computational perspective as well. An individual estimate of surprise for each agent may be intractable to obtain due to the non-stationarity of the environment. Instead, we seek minimize surprise jointly across all agents using an expressive Energy-based MIXer (EMIX) architecture which is compatible with any multi-agent RL algorithm. Figure illustrates our learning scheme.
Learning of surprise in the high-dimensional value function space is cumbersome with the number of actions scaling linearly in the number of agents. This imposes an inherent restriction to learn global surprise efficaciously across all agents a given timestep. Towards this goal. EMIX encodes individual value functions Q1, Q2, Qn corresponding to each agent using local value encoders. These encoders capture the local change in value functions arising over subsequent TD learning iterations (Wang et al., 10 A global state encoder maps environment states $1, 82, 8T to low dimensional representation. Further, state deviation encoder encodes deviations across all states $2: ... 8T within the given batch. Akin to a model-based method (Janner et al., 2019), the state deviation encoder accounts for uncertainty an agent's 's state visitation distribution. Note that the encoder does not construct an explicit model of states, but only represents their variation in the agent's 'sss environment. This insight essential to account for abrupt dynamics encountered by agents. Representations obtained from state and value function encoders are concatenated and compressed using a final surprise encoder which estimates a distribution of surprise values. The distribution implicitly represents the density of states wherein agent may encounter most surprise. value estimate Vaup& (s, u, o) sampled from the surprise distribution depicts the variational free energy configuration upon application of T which serves global intrinsic motivation. Practical training of EMIX proceeds with backpropagation (Rumelhart et al., 1986) using gradient descent and the reparameterization trick (Kingma & Welling, 2014) 4) for sampling of Va Veupp u,

---

Under review as a conference paper at ICLR 2022
4.4 PRACTICAL IMPLEMENTATION
Algorithm 1 Energy-based MIXer (EMIX)
1: Initialize @, 8, 0 ... fn agent and hypernetwork parameters.
2: Initialize learning rate a, temperature / and replay buffer R.
3: for environment step do
4: u + (u1, Ug.... u
5: R RU[(s r, s(}}
6:
if |R] > batch-size then for random batch do
7:
8:
Viat s
Q2.... O - QQ Q2 QN (7 = 1,2,,,,,,,,1
9:
10:
Calculate a and of using s and 8
11:
Kuurp (8, u, 0) Surprise-
u, a)
12:
surp (8 uul
Sringet-urrrise-Miiaee ,
EN exp (Vaurp (d/,af af (a'))
E E
Ed expp (Viu (s,4,a])
14:
Calculate L(0) using E in Equation 7
15:
+ 0 a oL((0)
16:
end for
17:
end
18:
if update-interval steps have passed then
19:
0, - 0, Vi 1,2,,,
end
21:
end for
Algorithm I presents the EMIX framework (in green) combined with QMIX Rashid et al. (2018), an off-the-shelf MARL algorithm. The total Q-value Qiat is computed by the mixer network with its inputs as the Q-values of all the agents conditioned on s via the hypernetworks. Similarly, the target mixers approximate Q conditioned on s'. In order to evaluate surprise within agents, compute the standard deviations a and o' across all observations z and for each agent using and s respectively. The surprise value function, called the Surprise-Mixer, estimates surprise V Viup (s. u, a) conditioned on s, u and a. The same computation is repeated using the Surget-Surprie-Miie estimating surprise Vie (s' u', a') within next-states in the batch. Application of the energy operator along the non-singleton agent dimension for Virrp(s (s, u, o)) and Va Viurp (5', u' , yields the energy ratio E which is used in Equation evaluate L()). We then batch gradient descent to update parameters of the mixer 0. Target parameters 0,, are updated every update intereal steps.
for
EXPERIMENTS
Our experiments aim to evaluate the theoretical claims presented by EMIX along with its performance to prior MARL methods. Specifically, we aim to answer the following questions- (1) How does the provision of an EBM for surprise minimization compare to current MARL methods?, and (2) Does the algorithm validate the theoretical claims corresponding to its components?
5.1 ENERGY-BASED SURPRISE MINIMIZATION
We assess the validity of EMIX, when combined with QMIX, multi-agent StarCraft II microman- agement scenarios (Samvelyan et al., 2019) as these consist of larger number of agents with different action spaces. This in turn motivates greater deal of coordination. Additionally, micromanagement scenarios StarCraft consist multiple opponents which introduce a greater degree of surprise within consecutive states.
We compare our method to prior methods namely; (1) QMIX (Rashid et al., 2018), constituting of nonlinear value function factorization with monotonicity constraints; (2) Value Decomposition Networks (VDN) (Sunchag et al., 2018), consisting of linear additive factorization of Q function; (3) Counterfactual Multi-Agent Policy Gradients (COMA) (Foerster et al., 2017), which consist of
7

---

Under review as a conference paper at ICLR 2022
Scenarios
EMIX
SMIRL-QMIX
QMIX
VDN
COMA
IQL.
2s. vs. Isc
90.33 = 0.72
88.41 1.31
89.19 = 3.23
91.42 + 1.23
96.90 +: 0.54
86.07 1: 0.98
2x3z.
95.40-10 0.45
94,93+0.32
95.30-14 1.28
92.03++2.08
43.33++2.70
55.74+6.84
3m
94,90-0. 0.39
93.94.10.22
93.43 0.20
94.58:10.58
84.75:17.93
94.79:10.50
3s.v8.22
99.58-0.07
97.63+1.088
99.43-0.20
97.90+-0.58
0.21+00.44
92.32+2283
3s.v8_42
97722-0.73
0.2410.11
96.01+3.933
94,2942.13
0.001:0.00
59.75112.22
38
11.91+11.80
0.00++0.00
43.44+7.09
68.51+5.60
0.00++0.00
18.14+2..4
3s5z
88.88=1.07
88.53+1.03
88,49=2.32
63.58:3.999
0,25+00111
7.05+3.52
8m
94.47_1.38
89,9611.42
94.3012.90
90,26.11.12
92.82.10.53
83.5311.62
8m 9m
71.03-2.69
69,90+1994
68,28+2.30
58.81+4.66
4,17+0.58
28.48+22..38
10m_vs.11m
75.35=2.30
77.85:2.02
70.36+2.87
76.81
4.55+:0.73
32.27 68
No.many_baneling baneling 5m_vs_6n
95.87
93.6110.94
93.35
92.26.11.06
91.65
1..97
37.07-2.42
33.2742.79
34.42+2.63
35.63++3.32
0.52+0.13
14.78+2.72
Table 1: Comparison success rate percentages between EMIX and prior MARL methods on
StarCraft II micromanagement scenarios. EMIX is comparable to improves over QMIX agent. In comparison SMiRL-QMIX, EMIX demonstrates improved minimization of surprise. Results are averaged over 5 random seeds.
counterfactual actor-critic updates in a centralized critic; and (4) Independent Q Learning (IQL) (Tan, 1993), wherein each agent acts independent of other agents. (5) In order to compare our surprise minimization scheme against pre-existing mechanisms, we compare EMIX additionally to a model-free implementation of SMiRL (Berseth et al., 2019) in QMIX. We use the generalized version of SMiRL as it demonstrates reduced variance across batches (Chen, 2020), This implementation is denoted as SMIRL-QMIX for comparisons. Details related to the implementation of EMIX are presented in Appendix D.
Table 1 presents the comparison of success rate percentages between EMIX and prior MARL algo- rithms on 12 StarCraft II micromanagement scenarios. Corresponding to each scenario, algorithms demonstrating higher success values in comparison other methods have their entries high- lighted in bold (see Appendix E.2 for a statistical analysis). Out of the 12 scenarios considered, EMIX presents higher success rates on 9 of these scenarios depicting the suitability of the proposed approach. In cases of so ro_many banelinnniing and 5m_vs .6m having large number of opponents and greater level of surprise, EMIX aptly improves over prior methods.
When compared to QMIX, EMIX depicts improved success rates on all of the 12 scenarios. On comparing EMIX with SMiRL-QNE -MIIX, we note that EMIX demonstrates a higher average success rate. This highlights the suitability of the mnergy-based scheme in the case of larger number of agents and complex environment dynamics for surprise minimization.
5.2 ABLATION STUDY
We now present the ablation study for the various components of EMIX. Our experiments aim to determine the effectiveness of the energy-based surprise minimization method. Additionally, also aim to evaluate the utility dual approximators for surprise estimation accordance with the precept from RL literature (Hasselt et al., 2016; Fujimoto et al., 2018; Haarnoja et al., 2018b).
5.2.1 EMIX ORJECTIVE
To weigh the effectiveness of energy- based scheme, we ablate the energy op- erator and only utilize Va suup. Since this implementation employs dual ap- proximators V (6) E (1,2) for sta- bility, we call this implementation as TwinQMIX. Thus, compare between QMIX. TwinQMIX and EMIX to assess the contributions of each of the proposed methods.
50, many baneling
Sm vs. 6m
Wall y
SICCENA nlt
2 U
QMX
Millon Scops
Million Stepp
Killion Staps
Figure 3: Ablations for each of EMIX's component. When compared QMIX, EMIX and TwinQMIX depict im- provements in performance and sample efficiency.
Figure 3 presents the comparison of aver- age success rates for QMIX, TwinQMIX and EMIX on different scenarios. Agents were evaluated for total of million timesteps with the lines in the plot indicating average success rates and the
8

---

Under review as a conference paper at ICLR 2022
shaded area the deviation across 5 random seeds. In comparison to QMIX, TwinQMIX adds stability to the original objective by incorporating surprising estimates in the initial QMIX objective. On comparing TwinQMIX to EMIX we note that dual approximators play little role in improving convergence. Thus, the energy-base surprise minimization scheme is the main facet for significant performance improvement in the modified EMIX objective. This is demonstrated in the 5m_vs_6m scenario wherein the EMIX implementation improves the performance of TwinQMIX in comparison QMIX by utilizing a surprise-robut policy. In the case of so many baneling scenario which consists of a large number of opponents (27 banelings), EMIX tackles surprise effectively by pre- venting significant drop in performance which is observed in cases of QMIX and TwinQMIX. We conjecture that this is direct consequence of underestimations arising from V (t) estimates.
5.2.2 SURPRISE MINIMIZATION WITH TEMPERATURE
The importance of 3 can be validated by assessing its usage in surprise minimization. However, is difficult to evaluate surprise minimization directly as surprise value function estimates V surp (s, a) vary from state-to state-to-state across different agents and thus, they present high variance during agent's learning. We instead observe the variation of E as it is a collection of surprise-bare based sample estimates across the batch. Additionally, E consists of prior samples Vauep (s, u, σ)) for Vaur (s', , u/ o') which makes inference across different agents tractable.
10th
many beraling
Figure 4: Variation of surprise minimization with temperature B. Learning of surprise is achieved by making use of suitable value of temperature parameter (8 0.01) which controls the stability in surprise minimization by utilizing E as intrinsic motivation.
Figure presents the variation of Energy ratio E with the temperature parameter during learning. We compare two stable variations of E at 3 0.001 and B 0.01. The objective minimizes E over the course of learning and attains thermal equilibrium with minimum energy. Intuitively, equilibrium corresponds to convergence to optimal policy x² which validates the claim in Theorem 2. With 3 = 0.01, EMIX presents improved convergence and surprise minimization for 5 out of the considered scenarios, hence validating the suitable choice of B. On the other hand, lower value of 33 0.001 does little to minimize surprise across agents.
6 DISCUSSION
In this paper, we presented an energy-based perspective towards surprise minimization in multi- agent RL. Towards this goal we introduce EMIX, an energy- intrinsic motivation framework for surprise minimization in MARL algorithms. EMIX utilizes a temporal EBM to estimate and minimize surprise jointly across all agents. Our theoretical claims on the formulation of minimization of temporal energy with surprise are corroborated upon utilizing EMIX on suite of challenging MARL tasks requiring significant collaboration under fast-paced dynamics.
While EMIX serves practical example of EBMs in cooperative MARL, it presents several new avenues for future work. We shed light on 3 such aspects.
(1) Provision of energy-based model naturally raises the question of how can we efficiently sample from the surprise distribution? Advances sampling methods depict promise towards this aspect.
(2) Although suitable for lower dimensions, the scalability of EBMs towards high dimensional action spaces remains an open question. We conjecture that the utility of density-based methods and generative models can address the scalability gap.
(3) Lastly, the extension of an EBM framework to opponent-awand and competitive MARL settings presents suitable tangent for learning multi-agent roles. Provision of an EBM for learning minimum energy role configurations would do away with the need for multi-stage training and complex exploration strategies. We leave the aforesaid as potential directions for future work.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning. 2017.
Kavosh Asadi and Michael Littman. An alternative softmax operator for reinforcement learning. In International Conference on Machine Learning, 2017.
Glen Berseth, Daniel Geng, Coline Devin, Dinesh Jayaraman, Chelsea Finn, and Sergey Levine. Smirl: Surprise minimizing rl in entropic enviromments, 2019.
Dimitri Bertsekas. Abstract dynamic programming. Athena Scientific Nashua, NH, USA, 2018.
Dimitri Bertsekas and John N Tsitsiklis. Neuro-dynamic Programming, volume 1. Athena Scientific, 1995.
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity-driven learning. In ICLR, 2019.
Lucian Buşoniu, Robert Babuška, and Bart De Schutter. Multi-agent reinforcement learning: An overview. In Innovations in multi-agent systems and applications-I. 2010.
Jerry Zikun Chen. Reinforcement learning generalization with surprise minimization, 2020.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients, 2017.
Karl Friston. The free-energy principle: a unified brain theory? Nature reviews neuroscience, 11(2): 127-138, 2010.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods, 2018.
Bolin Gao and Lacra Pavel. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint arXiv:1704.00805. 2017.
Jordi Grau-Moya, Felix Leibfried, and Haitham Bou-Ammar. Balancing two-player stochastic games with soft q-learning. arXiv preprint arXiv: 2018.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Advances in Neural Information Processing Systems 31. 2018.
Tuomas Haarnoja. Acquiring Diverse Robot Skills via Maximum Entropy Deep Reinforcement Learning. PhD thesis, UC Berkeley, 2018.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep ensrgy-based policies. arXiv preprint 170.v:1702.08165 2017.
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hicrarchical reinforcement learning. arXiv preprint arXiv:1804 02808, 2018a.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:180) 010190990 2018b.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski. Will Dabney, Dan Horgan, Bilal Piot. Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement leamnnn. arXiv preprint arXiv: 1710.0228,, 2017.
10

---

Under review as a conference paper at ICLR 2022
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, 2019.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model-based reinforcement learning for atari, 2019.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, 2014.
Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190, 02 2016.
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems, 2016.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. NeurlPS 2020, 2020.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on basey learning. Predicting structured data, 1, 2006.
Yann LeCun, Sumit Chopra, M Ranzato, and F-J Huang. Energy-based models in document recognition and computer vision. In Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), volume 1, 2007.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274, 2019.
Sergey Levine and Pieter Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems, 2014.
Michael A Lones. How to avoid machine learning pitfalls: guide for academic researchers. arXiv preprint Stti:2200.0029997 2021.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-competitiy environments, 2017.
Luis Macedo and Amilcar Cardoso. The role of surprise, curiosity and hunger on exploration of unknown environments populated with entities. In 2005 portuguese conference on artificial intelligence, 2005.
Luis Macedo, Rainer Reisezein, and Amilear Cardoso. Modeling forms of surprise in artificial agents: empirical and theoretical study of surprise functions. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 26. 2004.
David J. C. MacKay. Information Theory, Inference & Learning Algorithms. Cambridge University Press, 2002.
Henry B Mann and Donald R Whitney. On test of whether one of two random variables is stochastically larger than the other. Annals of Mathematical Statistics, 18, 1947.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, 2016.
John F. Nash. Equilibrium points in n-person games. Proceedings of the National Academy of Sciences, 36(1), 1950.
Jorge Nocedal and Stephen Wright. Numerical optimization. Springer Science & Business Media, 2006.
11

---

Under review as a conference paper at ICLR 2022
Brendan O' D'Dnoghue, Remi Munos, Koray Kavukcuoglu, and Volodymyr Mnih. Combining policy gradient and q-learning. arXiv preprint arXiv: 2016.
Frans A Olichoek and Christopher Amato, A concise introduction to decentralized POMDPs. Springer, 2016.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In ICML 2018: Proceedings of the Thirty- Fifth International Conference on Machine Learning, 2018.
Wei Ren, Randal W Beard, and Ella M Atkins. A survey of consensus problems in multi-agent coordination. In Proceedings of the 2005, American Control Conference, 2005., 2005.
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning Representations by Back-propagating Errors. Nature, 323:533-536, 1986.
Brian Sallans and Geoffrey E Hinton.
Reinforcement learning with factored states and actions.
Journal of Machine Learning Research, 5, 2004.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farguhar, Nantas Nardelli, Tim G. Rudner, Chia-Man Hung. Philip H. Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge, 2019.
John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q-learning. arXiv preprint arXiv: 1704.06440, 2017.
Philipp Schwartenbeck, Thomas FitzGerald, Ray Dolan, and Karl Friston. Exploration, novelty, surprise, and free energy minimization. Frontiers in psychology. 4:710, 2013.
Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv: 00814, 2015.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value-decompossttoon networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '1, pp. 2085-2087, 2018.
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the Tenth International Conference on Machine Learning, 1993.
Yee Whye Teh, Max Welling, Simon Osindero, and Geoffrey E Hinton. Energy-based models for sparse overcomplete representations. Journal of Machine Learning Research, 4, 2003.
Sebastian Thrun. Efficient exploration in reinforcement Icarning. 1992.
Marc Toussaint, Robot trajectory optimization using approximate inference. In Pruceedings of the 26th annual international conference on machine learning, 2009.
Oriol Vinyals, Igor Babuschkin, Wojciech Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John Agapiou, Max Jaderberg, and David Silver. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575, 11 2019.
Tonghan Wang. Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. [RODE}: Learning roles to decompose multi-agent tasks. In International Conference on Learning Representations, 2021.
Ermo Wei, Drew Wicke, David Freelan, and Sean Luke. Multiagent soft q-learning. arXiv preprint arXiv 2018.
Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning, arXiv preprint arXiv: 009007, 2019.
12

---

Under review as a conference paper at ICLR 2022
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010.
Brian D Ziebart, Andrew L Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI, 2008.
13

---

Under review as a conference paper at ICLR 2022
A PROOFS
sump (s, aa E N, the energy operator TV u, o) = log Ea=1 exp (Vsup(e, (s, u, σ)) forms contraction on Vaup (s, u, a).
Theorem 1. Given surprise value function V
Proof. We follow the process of (Asadi & Littman, 2017). Let us first define norm on surprise values ||V V2ll = max (s, a) V2(s, u, o)}. Suppose e = 1| 1211,
log
exp (8) I log Σ exp (V2
log
exp (Vi u, S log exp (c)
Σ
exp (V2 u.
N
log
Σ exp (8, S c + log Σ
exp (V2(8,
N
N
log
exp ()) - -log
exp (V2(5,u, S 1IV V/2l
(8)
a=1
Similarly, using with log E exp (V()),,,))),
N
N log Σ a=1
log
exp
exp (V2(8,
N
log exp (c)
exp (Vi(s,u,))) 2 log
exp (V)(),
a=1
o=1
log
exp (V((,,,,))) 2 log
exp (2(8,,,0))
||[ - 1/2l 2 log Σ exp (V2(s,u,)))
log Σ exp (Vi(s,
(9)
Results in Equation 8 and Equation prove that the energy operation is a contraction.
Theorem 2.
Upon agent's 's convergence to an optimal policy n", total energy of T", expressed by E* will reach a thermal equilibrium consisting of minimum surprise among consecutive states and s'.
Proof. We begin by initializing a set of M policies {TI, T.... JM} having energy ratios {,, Ez... EM }. Consider policy with surprise value function V1, Eg can then be expressed as
Σ N exp exp (v'(
E, log
Σi (vp((V(s. u.)
Invoking Assumption for and s', we can express (i u/ Vi 0) + G1 where Çi is a constant. Using this expression in Ei we get,
. exp (x Di (xp
F1 log
exp (G1 EN exp (V7(
E, log
Σm= exp (s,
Ej (1
14

---

Under review as a conference paper at ICLR 2022
Similarly, E2 (2,33 (3...,Es Cas. Thus, the energy residing in policy is proportional the surprise between consecutive states s and s'. Clearly, an optimal policy x' the one with minimum surprise. Mathematically.
42, TM = S I Ç1, S2..., SM Ex E1, Fe..., PM
m1; 22... TM
Thus, proving that the optimal policy consists of minimum surprise at thermal equilibrium.
B RELATION TO MAXIMUM ENTROPY FRAMEWORK
B.1 SIMILARITIES & DIFFERENCES
We conceptually compare EMIX the maximum entropy framework.
Both methods utilize an auxilary objective as intrinsic motivation to tackle uncertainty. While the maximum entropy formulation assigns low energy to uncertain actions, our method assigns low energy to uncertain encoded representations od states (as presented in Fig. 2).
Similarities:
Our method differs from maximum entropy its optimization process and learning scheme. The maximum entropy formulation aims to maximize entropy in the value function space so to motivate exploration. Our proposed scheme, the other hand, aims minimize surprise in the low-dimensional representation space to obtain dynamics-awarr aware robust policies.
Differences:
B.2 CONNECTION TO SOFT Q-LEARNING
The Soft Q-Learning objective with V (5') and Qsoft (u, 0) as state and action value functions (5')
respectively is given by-
[ (r (r+ FIlEss (s')] Qoort (u, s;
27
JQ(0) EE,.r~R
2
JQ) Eg g
r + TB log Σ exp Qoon
Qoont (u, 8; s;:)
2
The gradient of this objective can be expressed as-
VoJo(0)= BBauR ( {
log Σ exxp((ul.s';07)
Qooft (u, s;
V su
wEA
(10)
And the gradient of the EMIX objective is obtained as-
27
2a=1 exp (Vaurp (s', ul u'')) - Q(u, s;
L(0)= = Eg, S
r 7max + Blog
2
ΣN exp (Varp (s, U, σ))
VoL(O) = Ee,,,
r ymaxx(u'
Ea=1 exp (Va Veurp(s (s', ui, of)))
Q(..:10)) voQ((,,)0) (II)
+ 3 log
E exp (Var (s, u, σ))
Comparing Equation 10 to Equation 11 we notice that Soft Q-Learning and EMIX are related to each other as they utilize EBMs. Soft Q-Learning makes use of a discounted energy function which downweights the energy values over longer horizons. Actions consisting of lower energy configurations are given preference by making use of Qunit (u, s; 6) as the negative energy. On the other hand, EMIX makes of constant energy function weighed by which minimizes
15

---

Under review as a conference paper at ICLR 2022
crprise-based energy between consecutive states. Both the objectives can be thought of energy minimizing models which search for an optimal energy configuration. Soft Q-Learning searches for an optimal configuration in the action space whereas EMIX favours optimal behavior spurious states. In fact, EMIX can be realized as special case Soft Q-Learning the mixer agent utilizes energy-based policy and attains thermal equilibrium. This leads to express Theorem 3.
energy-based policy with its target function V log EneA exp Quu' s';0", the surprise minimization objective L(Ø) reduces to the Soft Q- Learning objective L(fuott) in the special case surprise absent between consecutive states, Ea=1 exp (V (Vspp('', (s',u', ",,,"o')) = = exp (Vaupl 1s, o)
Theorem 3. Given
Proof. We know that the EMIX objective is given by-
L(0) M
r ++ +++mxx((1ss00 log
a= exp
(Va surp ui, (urpp(',"',
1(u, s;
a= exp (Vsurp (s, u, σ))
(12)
Replacing the greedy policy term maxQ(u', with the energy-based value function V(s') log E''E exp Qu' s'; 07; we get,
Σ exp (Viup((s) (s', ut, uu',
L(0) = Eg s
+ 7. E [s'')) Blog
Q(u, s; 0)
Ex= exp (Vi (8, u, u.
(13)
1
L(0) Es,u. A
+ y.yy'
log
exp Q( (u''s';;"))
2
N'EA
N exp (Vaipp('' (8', (Viin('',
27 Quu, 0)
+ Blog
>a exp surp (s, U, σ))
At thermal equilibrium, Ex exp (V (8, o)) = Ed exp (s u',
1 L(0) RR
r + yEyyER log
Σ exp Quu' (';8)
2
n'En
N La=l exp (V (s', (d')""'''',')))
2
+ B log
Quu, 2(,,0;0) 8;
2a=1 exp (Vi Vuup (8', (8', u',
L(0) Eg
r log Σ exp spp' + Blog(1) Quu, s:0)
2
E'EA
(14)
27
1
L(() E.
+ Bo' log Σ exp ((p:(ul,s';07) W'E
(u,,s:
(15)
2
Equation 15 represents the Soft Q-Learning objective, hence proving the result.
C CONVERGENCE ANALYSIS
We now analyze convergence of the surprise minimization scheme during policy optimization. For brevity, our notation denotes the modified Bellman operator as B obeying the standard assumptions of monotonicity and contraction (Bertsekas, 2018). Additionally, we consider the cumulative value
V/2 Fж + G log En a exp(Va (arp. (4) (s, u, o)) as the sum of state values Vi г& G3 and surprise energy values / log EN exp Va (aurp.(k) (s, u, a)) at 1 Bellman update.
16

---

Under review as a conference paper at ICLR 2022
Consider
Vi - V*
with V* being the optimal value at convergence,
[/-- VO <
BV + Bog Σ exp V suTD.(s)) V
(16)
N
<
B² VV 2 Blog
exp(Vang surp, (k) + B log
exp(V V
surp.(k))
-V-
V*
(17)
a=1
o=1
2
N
B V/ 2 + <
log Σ
exp( W (krpp(k-1) + log
exp( V
(18)
aml
aml
B 1 + 3 <
log Σ exp( V
surpp((k-)) 11)
Σ exp( Va (impp./a)
V
(19)
Thus, for l iterations, we have,
1)
5 Vo s
log Σ (ump..)))
exp Va
V
(20)
1)
13th 3o +
log
exp
Va
su)p
V
(21)
a=
B + 3
log
exp
Va surp, (1)
V
(22)
We now absorb the sum of surprise values from time index i = 1, ... & in single variable Vi- Thus, sirp.(() and utilizing the Triangle Inequality, we get,
using Vos = Σ Va
VoV VO
| 2
+
3 log Σ (exp(Viee)
(23)
We now bound the two terms separately. Considering the first term and following the results of value iteration convergence (Bertsekas Tsitsiklis, 1995).
BVV V+
S 5* V
V
(24)
2
2
=gb v + Vi - V V
(25)
wherein Vú, denotes an approximation to V. Utilizing the triangle inequality yields,
Ve /|, +
v--
V/
(26)
The two terms are bounded using the convergence result of (Bertsekas, 2018).
mmax S
VTmax +
(27)
7
Now, considering the second term in Equation 23,
S log
Σ exp Va
(28)
N
N
3 log Σ
exp
Va - -og Σ
exp Viot) + log Σ exp Vi
(29)
a=1
17

---

Under review as a conference paper at ICLR 2022
using the triangle inequality,
N
N log
s 3 log Σ
exp((Ve.) - log Σ
exp
Viot
+
Σ exp(
V
(30)
a=1
Since log EN expl (ot) is a contraction following Theorem for the first term we have,
$ By
Vist
Vice
+
log exp (iot
Σ
(31)
a=1
The second term in the above relation is bounded due to the completeness assumption, log Ea- (Vit)
Br
Vioa
+ + <> 0
(32)
2 Finally, combining Equation 27 and Equation 32 in Equation 23, we obtain the desired convergence bound.
V/ |v--v-/ v/11 s 5* ( + V
S V
Fiax
+ B (1/ - -Vio +5)
(33)
While the first term in Equation 33 denotes the convergence of policy optimization, the second term indicates the bounded convergence of surprise to ecological niches with finite (yet nonzero) surprising elements. The policy optimization process converges a geometric rate ((x*) towards its stable fixed points. The surprise minimization process, the other hand, demonstrates annealing behavior which depends on the temperature parameter B. Furthermore, convergence to stable fixed point V is bounded in respect to each agents individual surprise values V This insight indicates that different agents converge towards different locally optimal values of surprise. Finally, the presence of constant Ç corroborates prior claims (Schwartenbeck al., 2013; Friston, 2010) that agents continue to experience surprise irrespective of their convergence to minimum energy niches.
To further develop intuition for this claim, consider the special case wherein
Via
V
Irrespective of global convergence among all agents, finite yet small Ç continues contribute the upper bound of V
V'
Role of We further discuss the role of |3 which is of balancing the terms at successive iterations. While the first term geometrically decays with O(**) ) rate, the second term approaches finite constant BC as V + Vi Irrespective of our choice of J3, the LHS V V* is upper bounded by constant which validates the claims of minimum yet finite surprise values. We do note that small / is still desirable to remove any approximation errors in order to push V22 + V*, However, this comes at the cost of increased surprise if (3 is not selected appropriately.
D IMPLEMENTATION DETAILS
D.1 MODEL SPECIFICATIONS
This section highlights model architecture for the surprise value function. At the lower level, the architecture consists of 3 independent networks called sette_net, qenet and surp_net. Each of these networks consist of a single layer of 256 units with ReLU non non-linearity as activations. Similar to the mixer-network, we use the ReL.U non-linearity in order to provide monotonicity constraints across agents. Using modular architecture in combination with independent networks leads to richer extraction joint latent transition space. Outputs from each of the networks are concatenated and are provided input to the main consisting of 256 units with ReLU activations. The main _ee yields single output as the surprise value Vaup(( (s, u, σ which reduced along the agent dimension by the energy operator. Alternatively, deeper versions of networks be used in order to make the extracted embeddings increasingly expressive. However, increasing the number of layers does little in comparison to additional computational expense.
Architecture:
18

---

Under review as a conference paper at ICLR 2022
Computation of The deviation corresponds to the standard deviation across each dimension of the state 8. Considering the state as a tensor of size B x x M with B as the batch size, A as the number of agents and M as the observation dimension, deviation across the M dimension. This yields o as a B x x dimensional array.
compute by calculating the standard
Vsurp denotes the surprise value function which quantifies the amount of surprise experienced by agents. Analogous to a Q value function which provides estimates of returns, Vaurp provides estimate of surprise. Our framework learns Vearp much like any other value function (using a neural network), but by additionally undergoing a log Σ exp transformation to obey the fixed point property. This is achieved by realizing log-sum sum-exp an energy operator T = log Σ exp which can be computed using standard computation libraries. Since our code is im- plemented in PyTorch, we implement this T_V torch. Logstmexp (V_surp, dim=1).
Computation of surprise estimates:
D.2 HYPERPARAMETERS
Table 2 presents hyperparameter values for EMIX. A total of 2 target Q-functions were used as the model is found to be robust to any greater values.
Hyperparameters
Values
batch size learning rate discount factor target update interval gradient clipping exploration schedule mixer embedding size agent hidden size temperature target Q-functions
b 32
= 0.0005
? 0.99 200 episodes
10
1.0 to 0.01 over 50000 steps
32
64
B = 0.01
Table 2: Hyperparameter values for EMIX agents
D.3. SELECTION & TUNING OF
One can manually tune 3 using a fine- fine-grained hyperparameter search. We tune between 0.001 and 1 in intervals of 0.01 with best performance observed at B = 0.01. However, we find two additional methods helpful for obtaining more accurate values. These are described as follows-
Armijo's 's Line Search: One can borrow from optimization theory and utilize Armijo's 's line search Nocedal & Wright (2006) by setting a termination condition. The method starts with a constant value of 3 which is iteratively incrementeddddecreeenttd until termination criterion (example- [[V]( < with e a constant) is reached. While line search is proven to converge towards globally optimal values, its Onz²) convergence may be computationally expensive that too in the MARL setting. Thus, we turn to the more efficient automatic tuning.
Algorithm 2 Armijo's Line Search
Algorithm Automatic Tuning
1: Initialize 3, E (0, 1). EMIX & TVo surp
1: Initialize 3, e (0, EMIX & TV surp
2: while EMIX(Q + 3 Va surp) > EMIX(Q)
2: EMIX(Q + 3 TVV (yurp))
+ O * 3* VEMIX(Q)*TV
surp do
beta_loss 3 0.5 (TVe VTVu 0)2
3:
8= 6==6 8 3
4: beta_loss. backwbaac
4: end while
return
5: return
Automatic Tuning: We choose to automatically tune 3 following single-agent RL literature Haarnoja et al. (2018b); Kumar al. (2020). This is achieved by treating & parameter and adaptively
19

---

Under review as a conference paper at ICLR 2022
optimizing over it using Adam. We treat surprise value of 0 as our target value. The method works well in practice and provides /3 values closer to 0.01 (our manual selection).
E ADDITIONAL. RESULTS
E.1 QUALITATIVE ANALYSIS
Figure 5: Task- sanmaan__aaaaaae baneling, (left) Behaviors learned by EMIX agents, (right) Behaviors
learned by QMIX agents
a RT
Figure 6: Task- 2s vs Isc, (left) Behaviors learned by EMIX agents, (right) Behaviors learned by QMIX agents
We visualize and compare behaviors learned by surprise minimizing agents to the prior method of QMIX. Fig. presents the comparison of EMIX and QMIX agent trajectories (in yellow arrows) on the challenging so many baneling task. The task consists of 27 baneling opponents which rapidly attack the agent team on a bridge. QMIX agents naively move to the central alley of the bridge and start attacking enemies early on. While QMIX agents naively maximize returns, EMIX agents learn different strategy. EMIX agents rearrange themselves first at the corners of the bridge. Note that these corners provide cover from enemy's 's fire. Thus, EMIX agents learn to take cover before approaching the enemy head-on. This indicates that the surprise-robust policy is aware of the incoming fast-paces assault.
As another example, Fig. 6 presents behaviors on the 2s_vs. task wherein two agents must collaborate together to defeat SpineCrawler enemy. The enemy, having long tentacle pointing to the front, chooses to attack any one of the agents randomly in front of it. Additionally, the tentacle has a fixed length and cannot extend beyond this range. Random intermittent attacks indicate that the agents face greater degree of surprise with no prior knowledge of the enemy's movement. We observe that QMIX agents take turns to attack the enemy by moving back and forth to minimize damage. EMIX agents, on the other hand, learn different strategy. One of the EMIX agents stands at a distance to attack th enemy while the other agent goes around to attack from behind. This indicates that the policy is aware of enemy's limited movement.
E.2 STATISTICAL SIGNIFICANCE
We follow the recommendation of Lones (2021) and evaluate the statistical significance of our results by carrying out the Mann- WhinnWhitneyy U test Mann & Whitney (1947). All 5 seeds of an algorithm
20

---

Under review as a conference paper at ICLR 2022
Scenarios
EMIX
SMiRL-QMIT
QMIX
VDN
COMA
IQL
2s_vs_Isc
14
7
21
25
4
2s37
15
9
6
0
0
3m
17
0
0
2
12
3s_vs_3%
11
3
0
1
3s_vs_4z 4z
21
0
2
0
3s_vs_57
5
0
25
0
0
3s5z
7
13
0
0
8m
15
1
3
0
8m_vs_ inn _m_
7
11
0
0
0
10m_vs_1
14
25
0
0
so no.many_bancling
24
14
0
5m_vs_6m
21
15
18
Table 3: Comparison of the 24 statistic on StarCraft II benchmark, U here denotes the statistical
significance of algorithm against QMIX (higher is better).
(on each task) are compared to that QMIX to yield the 2 statistic. 2 here denotes the statistical significance of performance with higher values being desirable.
Table 3 presents the comparison
statistic on the StartCraft II benchmark. EMIX demonstrates consistently high values of uI across diverse set of tasks when compared to SMiRL and prior MARL agents. This highlights the consistent surprise-ming minimizing performance EMIX across random seeds.
E.3 ADDITIONAL TASKS
This section compares EMIX and TwinQMIX to prior MARL methods on the Predator-Prey tasks, In addition to the difficulty of task, vary the number of opponents. This helps quantify the variation in performance against increasing level surprise under fixed dynamics. Table 4 presents average returns. While all agents present comparable performance the easier tasks, EMIX improves QMIX and TwinQMIX on the more challenging punish and hard tasks. In the case of punish, EMIX is the only method to achieve greater than 20 returns outperforming baselines by significant margin.
Sccuarios
EMIX
TwinQMIX
SMIRL-QMIX
QMIX
VDN
COMA
[QL
predator_prey
40.00 1. 0.13
40.00 1. 0.34
40.00 0.98
40.00 1: 0.22
38.74 1: 0.64
27.49 1 4.26
34.73
2.92
predator, prey
predator_pree punish
24.17 1: 3.29
20.32 1 4.15
19.31
1.12
14.33 1: 3.81
17.21 4: 2.31
10.92
1 4.35
7,86 :: 3,21
prodator, prey
+
10.47
8.76 4.33
5.19
-4.37
1.53
.9.26
Table 4: Comparison of average returns between EMIX, its ablations and prior MARL methods on Predator-Prey tasks. EMIX improves over QMIX agent. In comparison SMiRL-QMIX, EMIX demonstrates improved minimization of surprise. Results are averaged over 5 random seeds.
We consider simple toy task from the Predator-Prey benchmark to demonstrate the importance of surprise minimization. We select preda- tor tor-prey_easy due to its simplicity and convenient dynamics. The task consists of agents and 3 opponents. We increase the number of opponents while keeping the task fixed. This way the dynamics of the MDP remain unchanged and the only changing factor is opponent behaviors.
Fig. 7 presents the variation of average returns for EMIX and QMIX over 5 random seeds. While QMIX agents undergo a steady decrease performance, EMIX agents are found robust to this fast degradation. Even after the addition of 20 opponents (against only 3 agents), EMIX is able to retain positive returns. The algorithm acquires surprise robust-policy early on during training to tackle fast-paced changes introduced by the large number of agents.
Nmmbe of Opoocrete
Figure 7: Variation in performance with increasing number of agents.
21

---

Under review as a conference paper at ICLR 2022
E.4
ADDITIONAL ABLATIONS
35
Bm
5m 6m
Whs knw mn Mar e an foll
Success rate %
Success rate %%
Surcess rateo
0.1
0.01
0.001
Million Steps
Million Staps
Million Steps
Figure 8: Variation in success rates with temperature B. A value of 0.00 is found to work best.
22


Reviews:
[{"review_id": "tb9_8OJBKaQ", "reviewer": "Reviewer_g9cM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n--The presentation is clear and easy to follow. \n--The empirical results and ablation studies are sufficient.\n\n\nWeaknesses (Questions):\n\n1- The ``deviation $\\sigma$ within states for each agent $a$ ” is not clear to me. As the authors only defined observations $O$ for each agent in Sec 3.1, is  $\\sigma$ the deviation within all agents’ observations?\n\n\n2- in Eq. (7), should the second $Q(u,s;θ) $ read as $\\hat{Q}(u,s;θ)$?\n\n3- Eq.(3) is not clear to me. The surprise is also not defined yet. Why should Eq(3) represent a global surprise? How minimizing Eq(7) can minimise surprise? \n\n4- Table 1 shows that EMIX improves the winning rate in most of the considered tasks. But it seems the improvement is marginal. This is also the case for the results in Figure 3.\n\n5- Only using SC-II micromanagement is not convincing enough. It is better to include results on other different benchmarks.\n", "summary_of_the_paper": "This paper focuses on the value factorization approach for solving multi-agent problems. It  considered the spurious surprise states in execution and proposed to address this problem through the proposed free-energy minimization framework.\n", "summary_of_the_review": "\nThis paper is attempting to extend the surprise minimisation to the multi-agent learning regime. My major concern is that Eq.(4) seems to be solely a marginally new way of representing the Q-function. It is not different from the typical Q-Mix or Q learning. \n\nThe core message of how Eq(4) or Eq(7) can minimise surprise is not clear to me either.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Gsv2tgnUdTY", "reviewer": "Reviewer_doix", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work builds in the interesting new directions around surprise minimization and energy based models.  Moreover, it provides a powerful theoretical framework incorporating ideas from energy based models and providing an algorithm with convergence grantees.  To the best of my knowledge the mathematical results seem correct, though I have not checked the work in the Appendix, they are intuitive and seem to be extensions of existing ideas.  \n\nThough the theoretical results are strong, the empirical results are not as compelling.  For an evaluation of this method, I would expect some sort of qualitative evaluation showing that the agents in fact behave in a way to minimize surprise.  The agents on their own should show the same qualitative behaviors seen in surprise minimization in single agent settings, but, more importantly, it is important to be clear about the qualitative differences we should expect in multi-agent settings and to verify that those effects occur since there is added complexity with the multi-agent aspects of the problem. \n\nWithout those qualitative effects, empirically the results seem to be marginal, for instance in Table 1 it is difficult to tell which of the results are statistically significant.  It seems like most of the error bars overlap, so the results are statistically insignificant, but only one of the methods is bolded.  To be clear, I do not think that the method needs to surpass all others on success rate, it isn't designed to make those sorts of improvements, but because that is all we have for empirical evaluation the best I can conclude is that the intervention into QMIX does not break the method.\n\nOn a related note, I think the work would be much more intuitive if the utility of minimizing surprise in multi-agent RL, and the unique challenges that come with multi-agent RL were made more clear from the beginning.  As it stands, I understand from the introduction I understand that EBM are a natural way to approach surprise minimization in multi-agent RL, but it isn't clear to me why you would want surprise minimization in multi-agent RL particularly, and what is different about multi-agent RL that would effect the sorts of behavior we would expect out of surprise minimization.  I think this could be alleviated by an example showing how minimizing surprise in MARL produces desirable effects, and that doing surprise minimization without considering any multi-agent aspects does not produce that desirable behavior.\n\n\nMinor point:\n* In the final paragraph of page 3, the Q_a is said to be lower bounded by Q^* when it is demonstrated to be upper bounded by Q^*", "summary_of_the_paper": "This paper proposes a method in the setting of centralized training and decentralized execution for training policies which minimize surprise.  The method uses insights from energy based models, deriving an energy operator that forms a contraction operator on the appropriate value functions, and uses this to make a variant of QMIX, which promotes surprise minimization across the multi-agent system.", "summary_of_the_review": "I will weakly recommend rejection due to the limited qualitative evaluation and otherwise inconclusive empirical results.  I would be willing to reconsider my score if the approach could be show to promote the expected qualitative effects of minimizing surprise in a simple setting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GsZUb1JdWmF", "reviewer": "Reviewer_zFND", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There is, what appears to be, a very interesting argument being made here and it draws together a number of strands of research from different disciplines. The multi-agent domain being a clear candidate where independent gradient ascent from individual agents can result in a myopic trap interfering with convergence to a global objective. Nevertheless, a number of the ideas, methods, equations and results in this paper are not as clear to me as I would like and so I cannot endorse the paper for acceptance.\n\nOne of the main themes that was running through the paper was the idea of surprise minimisation and how this has some degree of tension with the surprise maximisation aspect of maximum entropy (which as the authors say is a strategy for ensuring good exploration). However, this tension is never properly resolved to my satisfaction. \n\nIt is clear that the results show advantages over the other SoTA methods, but I am not sure that the method could be replicated by a reader and so I am unsure whether these experiments could be validated.\n\n## Detailed notes\n\nThe term EBM needs defining when first used.\n\nShould really cite the Dec-POMDP literature rather than the POMDP paper.\n\np2-3: Is  the joint action a cross product over identical action spaces? Or can each agent have different actions available.\n\nIf the observation function is a function (determined) then do all agents get the same observation at time t? Or is it probabilistic but the same distribution for all agents? Or does/can it depend on the agent?\n\nThe policy and value function seems to depend on the hidden state.\n\nI think you mean that $\\pi^\\star_\\theta = \\pi_{\\theta^\\star}$ (notationall that the best policy. It's clear what was intended but the notation is sloppy.\n\nIn equation 1, b is not very clear. Is this the expectation of the sample average over the minibatch? In which case that is not what the equation says. In short, b should enter into the equation somewhere other than just the subscript to the expectation. Also, sampled from memory R is ill-defined.\n\nFigure 1 does not clearly illustrate what it intends to. There is no description of what the grey surface is, nor of the blue points on this surface and the connecting lines. Yes, the little pictures of robots with a red background are associated with a region of lighter grey, and the green associated with darker grey (which we are told is associated with energy minima), but in all this doesn't leave the reader any the wiser as to what is going on.\n\nOne of the arguments to the surprise value function $V_{\\text{surp}}$ is $\\sigma$ which is described as \"deviation...within states for each agent a\", but it isn't clear what is meant by this.\n\nThe function $V_{\\text{surp}}$ is not clear at all. It is said that the $\\Tau$ operator is a log-sum-exp contraction for this function but it isn't clear how to calculate this in the first place. Maybe I am missing something.", "summary_of_the_paper": "The authors present a method to regularise the learning of Q values within Decentralised partially observable markov decision processes, where the regulariser is one that minimises surprise in some way across the population of agents within the environment.\n\nThis it is argued allows the agents to avoid situations in which states are \"rapidly changing\", instead aiming to reach an equilibrium state where just enough surprise is being experienced as part of a reward maximising objective.\n\nThe authors present a series of results in which their method outperforms a number of SoTA alternatives on a reasonable looking set of benchmarks, as well as an ablation study showing the criticality of each proposed component.\n", "summary_of_the_review": "I confess that I miss the point of the paper. I am not confident that I could recreate the method and this potentially of by-product of key bits of information being excluded from explanations. I am fairly confident that the authors have a strong technical understanding and justification for what they have done, but this is not expressed clearly in the paper, at least not in a way that I can decode.\n\n## Update after rebuttal\n\nMy concerns have been partly addressed by clarification on what measures of surprise can be used and how these would be estimated. The publication of the code also goes some way to addressing my concerns. The other updates also improve the paper and the explanations and so I am changing my recommendation to weak accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "49MFWw6P4Bp", "reviewer": "Reviewer_8Wiu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n1. The writing of this paper is clear enough and the presentation manner is concise.\n2. The proposed method is novel to some extent.\n3. The comparison between the proposed method and other related works is good.\n4. Ablation studies are conducted to demonstrate the effects of hyperparameters.\n\n## Weaknesses\n\n1. Although the authors claimed that the proposed learning framework is for MARL, the whole theoretical framework is weakly connected to the context of MARL (in decentralized manner). The only connection is the assumption that the sum of suprising values instead of the single global surprising value (that can be seen as a centralized optimization that is equivalent to the single agent case). In my view, it could be more appropriate to write this paper as a single agent RL work and the contributions would not be reduced. For now, this work is actually a theory based on the single agent scenario and an trivial extension to the multi-agent scenario given the assumption of the global suprising value being equal to the sum of decentralized surprising values.\n2. As the authors said in Theorem 3, the proposed surprising value objective can be reduced to the soft Q-learning objective. Could the authors explain in more details why the the proposed surprising value objective is necessary? The biggest concern from me is that it still needs to learn the thermal equilibrium of the surprising value. In other words, during the learning process the proposed surprising value is **not** constant, which may conflict with the claims in the paper.\n3. In the concergence analysis, the authors subtly divided the suprising terms and the orginal value terms so that these two terms can be proved separately. **Nevertheless**, there exists a potential issue that the time scales of these two convergences (maybe inaccurate) are actually inconsistent. Specifically, the oprator of the orginal values is with respect to the dynamic environment, so it is corresponding to the transition between different states. While the oprator of the surprising value is actually for the static scenario (if I understand correctly), i.e., for a given state-action pair. Therefore, it could lead to a potential issue that the rate $\\beta$ for the operator of surprising value may impact the convergence of optimal policies. Could the authors discuss more about this issue? The convergence rate of the whole sequence needs to be discussed in my view, according to the $\\beta$.\n4. The authors have compared the proposed algorithm with MARL algorithms, but most of them are irrelevant to the surprise term or intrinsic reward (as the authors claimed the relation in the paper). Can the authors show more comparisons with these highly relevant baselines so as to show the effectiveness of the proposed method?\n5. About the SMAC experiment, some of the experimental results shown in the paper are far lower than the existing results, e.g. 5m\\_vs\\_6m, 3s\\_vs\\_5z, 10m\\_vs\\_11m. Can the authors give a convinciable reason or rerun these results by the latest baseline implementation from Pymarl.\n6. As I concerned in theoretical analysis above, the choice of $\\beta$ is important that is demonstrated in the abalation study. Can the authors show some laws or insights on how to select $\\beta$?", "summary_of_the_paper": "This paper introduced a suprise term in optimizing the policies (action-value functions in Q-learning) for solving the non-stationary challenge due to the rapid changes from the environment for MARL scenarios. This work not only proposed the concept of suprising value in the context of MARL, but also gave an operator (i.e. a contration) that can make the sequence of suprising values converges to a fixed point. The general sketch of the proofs is correct. Also, the authors showed that the convex conjugate of the suprising value operator is analogous to the minimization of the uncertainty among agents. By incorporating the suprising value into the Q-learning algorithm, the term led by the ratio between target and behaviour suprising value (suprising ratio) can also be interpreted as an intrinsic reward. Moreover, the authors also compare the proposed surprising minimization objective with the Soft Q-learning objective. ", "summary_of_the_review": "In summary, this work proposes an interesting problem and give a thourough analysis on the proposed method. However, due to the multiple concerns on the theoretical and experimental results, at the moment I can only recommend reject (but marginally below the threshold).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper explores surprise minimization in multi-agent learning by using free energy across all agents in a multi-agent system. A temporal EBM represents an estimate of surprise which is minimized over the joint agent distribution. Empirical studies on the proposed method are conducted. This paper builds in an interesting direction around surprise minimization in multi-agent learning by using the energy-based framework, but the presentation of the method seems to need more efforts to be improved to avoid confusion. 

The discussion between authors and reviewers is summarized below: The major concerns of Reviewer doix include that: (i) the empirical results are not compelling, (ii) qualitative results are missing, and (iii) the motivation of surprise minimization in multi-agent RL is unclear. After the rebuttal, the authors addressed the concerns of Reviewer doix, who changed his/her score from 5 to 6. The major concern of Reviewer zFND comes from the understanding and justification of the paper. After the rebuttal, the concerns of Reviewer zFND  have been partially addressed by clarification on what measures of surprise can be used and how these would be estimated. Reviewer zFND eventually changed his/her score from 5 to 6.  Also, most of the concerns about theory and experiments from Reviewer 8Wiu have been addressed after the rebuttal. Reviewer 8Wiu accordingly changed the rating from 5 to 6. Reviewer g9cM is still not satisfied with the authors' answers, and his/her concerns regarding some technical issues remain and points out that the current paper has many inconsistencies across the writing that make it hard to evaluate the soundness and correctness of the results.  

After the rebuttal, the author successfully addressed most of the concerns from 3 of 4 reviewers, but the overall rating of the paper is on a borderline level. Given the fact that the paper still has some unaddressed concerns from Reviewer g9cM, and other reviewers actually do not champion the paper. The AC tends to recommend rejecting the paper at the current stage. AC urges the authors to improve their paper by including all the suggestions provided by the reviewers, and then resubmit it to a future venue.
Decision: Reject