Efficient Episodic Memory Utilization of Cooperative Multi-Agent Reinforcement Learning
Authors: Hyungho Na
Yunkyeong Seo
Il-chul Moon
Venue: ICLR 2024 oral (2024)
Keywords: Multi-agent reinforcement learning
episodic control
episodic incentive
state embedding
Primary area: reinforcement learning
TL;DR: We introduce a framework that enhances memory utilization in cooperative multi-agent reinforcement learning to achieve a common goal through semantic embedding and episodic incentives.

Abstract: In cooperative multi-agent reinforcement learning (MARL), agents aim to achieve a common goal, such as defeating enemies or scoring a goal. Existing MARL algorithms are effective but still require significant learning time and often get trapped in local optima by complex tasks, subsequently failing to discover a goal-reaching policy. To address this, we introduce Efficient episodic Memory Utilization (EMU) for MARL, with two primary objectives: (a) accelerating reinforcement learning by leveraging semantically coherent memory from an episodic buffer and (b) selectively promoting desirable transitions to prevent local convergence. To achieve (a), EMU incorporates a trainable encoder/decoder structure alongside MARL, creating coherent memory embeddings that facilitate exploratory memory recall. To achieve (b), EMU introduces a novel reward structure called episodic incentive based on the desirability of states. This reward improves the TD target in Q-learning and acts as an additional incentive for desirable transitions. We provide theoretical support for the proposed incentive and demonstrate the effectiveness of EMU compared to conventional episodic control. The proposed method is evaluated in StarCraft II and Google Research Football, and empirical results indicate further performance improvement over state-of-the-art methods.

Published as a conference paper at ICLR 2024
EFFICIENT EPISODIC MEMORY UTILIZATION OF COOP- ERATIVE MULTI-AGENT REINFORCEMENT LEARNING
Hyungho Na', Yunkyeong Seo' & Il-Chul Moon
"Korea Advanced Institute of Science and Technology (KAIST), "summary.ai ggddgh233 @ggmall. com, (t jabsrud, icmoon @kaist. ac. kr
ABSTRACT
In cooperative multi-agent reinforcement learning (MARL), agents aim to achieve common goal, such as defeating enemies or scoring a goal. Existing MARL algo- rithms are effective but still require significant learning time and often get trapped in local optima by complex tasks, subsequently failing discover goal-reaching policy. To address this, we introduce Efficient episodic Memory Utilization (EMU) for MARL, with two primary objectives: (a) accelerating reinforcement learning by leveraging semantically coherent memory from an episodic buffer and (b) selec- tively promoting desirable transitions prevent local convergence. To achieve (a), EMU incorporates a trainable encoder/decoder structure alongside MARL, creating coherent memory embeddings that facilitate exploratory memory recall. To achieve (b), EMU introduces novel reward structure called episodic incentive based on the desirability of states. This reward improves the TD target in Q-learning and acts additional incentive for desirable transitions. We provide theoretical support for the proposed incentive and demonstrate the effectiveness of EMU compared to conventional episodic control. The proposed method is evaluated in StarCraft II and Google Research Football, and empirical results indicate further performance improvement over state-of -frtthrart methods. Our code is available at: https: //gi thub. com/ homm/Huuggoooaa EMU.
INTRODUCTION
Recently, cooperative MARI. has been adopted to many applications, including traffic control (Wier ing et al. 2000), resource allocation (Dandanov et al. (017)), robot path planning (Wang et al. 2020aa. and production systems (Dittrich & Fohlmeister 2020), etc. In spite of these successful applications, cooperative MARL still faces challenges in learning proper coordination among multiple agents because of the partial observability and the interaction between agents during training.
To address these challenges, the framework of centralized training and decentralized execution (CTDE) (Olichoek et al.) 2008: Olichoek & Amato, 2016 Gupta et al. 2017) has been proposed. CTDE enables decentralized execution while fully utilizing global information during centralized training, so CTDE improves policy learning by accessing to global states at the training phase. Especially, value factorization approaches (Sunchag et al. [0177] Rashid al. 20188) Son et al. 2019 Yang et al. 2020 Rashid et al. [2000] Wang et al. (020bb maintain the consistency between individual and joint action selection, achieving the othe-rt performance on difficult multi-agent tasks, such as StarCraft II Multi-agent Challenge (SMAC) (Samvelyan et al. 2019). However, learning optimal policy in MARL still requires a long convergence time due the interaction between agents, and the trained models often fall into local optima, particularly when agents perform complex tasks (Mahajan et al., 2019). Hence, researchers present committed exploration mechanism under this CTDE training practice (Mahajan et al. 2019 Yang et al. 2019 Wang et al. 2019 Liu al. 1021 with the expectation to find episodes escaping from the local optima.
Despite the required exploration in MARL with CTDE, recent works on episodic control emphasize the exploitation of episodic memory to expedite reinforcement learning. Episodic control (Lengyel & Dayan 2007| Blundell et al.) 2016 Lin et al.] 2018: Pritzel al.) (017) memorizes explored states and their best returns from experience in the episodic memory, to converge on the best policy. Recently, this episodic control has been adopted to MARI. (Zheng et al. 2021 and this episodic

---

Published as a conference paper at ICLR 2024
control case shows faster convergence than the learning without such memory. Whereas there merits from episodic memory and control from its utilization, there exists a problem of determining which memories to recall and how to them, to efficiently explore from the memory. According to Blundell et al. (2016); Lin et al. (2018): Zheng et al. (2021), the previous episodic control generally utilizes a random projection
embed global states, but this random projection hardly makes the semantically similar states close to one another in the embedding space. In this case, exploration will be limited
narrow distance threshold. However, this small threshold leads to inefficient memory utilization because the recall of episodic memory under such small thresholds retrieves only the same state without consideration of semantic similarity from the perspective of goal achievement. Additionally, the naive utilization of episodic control on complex tasks involves the risk of converging to local optima by repeatedly revisiting previously explored states, favoring exploitation over exploration.
This paper presents an Efficient episodic Memory Utilization for multi-agent rein- forcement learning (EMU), a framework to selectively encourage desirable transitions with semantic memory embeddings.
Contribution.
Efficient memory embedding: When generating features of a global state for episodic memory (Figure[[ we adopt encoder/decoder structure where 1) an encoder embeds a global state conditioned on timestep into a low-dimensional feature and 2) a decoder takes this feature as an input conditioned on the timestep to predict the return of the global state. In addition, to ensure smoother embedding space, we also consider the reconstruction of the global state when training the decoder to predict its return. To this end, we develop deterministic Conditional AutoEncoder (dCAE) [ With this structure, important features for overall return can be captured in the embedding space. The proposed embedding contains semantic meaning and thus guarantees a gradual change of feature space, making the further exploration on memory space near the given state, i.e., efficient memory utilization.
Episodic incentive generation: While the semantic embedding provides a space explore, we still need to identify promising state transitions to explore. Therefore, we define a desirable trajectory representing the highest return path, such as destroying all enemies in SMAC or scoring goal in Google Research Football (GRF) (Kurach et al.. (020). States on this trajectory are marked as desirable in episodic memory, so we could incentivize the exploration on such states according to their desirability. We name this incentive structure as an episodic incentive (Figure [()).. encouraging desirable transitions and preventing convergence to unsatisfactory local optima. We provide theoretical analyses demonstrating that this episodic incentive yields a better gradient signal compared to conventional episodic control.
We evaluate EMU on SMAC and GRE, and empirical results demonstrate that the proposed method achieves further performance improvement compared to the state-of-art baseline methods. Ablation studies and qualitative analyses validate the propositions made by this paper.
PRELIMINARY
2.1 DECENTRALIZED POMDP
A fully cooperative multi-agent task can be formalized by following the Decentralized Par- tially Observable Markov Decision Process (Dec-POMDP) (Oliehoek & Amato, 2016). G = S. R. Ω O, n, 7). where I is the finite set of n agents; E S the true state of the environment; 0g E A is the i-th agent's action forming the joint action a E A"; P(s'|s, a) is the state transition function: is a reward function is the observation function generating an observation for each agent E Ω; and finally, 7 E (0, 1) is a discount factor. At each timestep, an agent has its own local observation on, and the agent selects an action a; E A. The current state and the joint action of all agents lead to next state s' according to P(s'|s, a). The joint variable of s, a. and will determine the identical reward r across the multi-agent group. In addition, similar to Hausknecht & Stone (2015); Rashid et al. (2018), each agent utilizes a local action-observee observation history T, E T = (52 x A) for its policy n. a T.), where
R(s, a. E R; 92 the observation space; O
x A >> [0, 1).
2

---

Published as a conference paper at ICLR 2024
2.2 DESIRABILITY AND DESIRABLE TRAJECTORY
Definition 1. (Desirability and Desirable Trajectory) For a given threshold return Rear and a trajectory T : {80, ao, ro, 81, 01; 71, .T,)), J. T is considered as desirable trajectory, denoted Tg. when an episodic return is R ET-Irg ry 2 Rin- A binary indicator {(-) denotes the desirability of state as (sa) = 1 when E VTg
In cooperative MARL tasks, such as SMAC and GRE, the total amount of rewards from the environ- ment within an episode is often limited as Rmax. which is only given when cooperative agents achieve a common goal. In such a case, we can set Rehr = Rmax- For further description of cooperative MARL, please see Appendix A
2.3 EPISODIC CONTROL IN MARL
Episodic control was introduced from the analogy of a brain's hippocampus for memory utilization (Lengyel & Dayan 2007). After the introduction of deep Q-network, Blundell et al. (2016) adopted this Idea of episodic control to the model-free setting by storing the highest return of given state, to efficiently estimate the Q-values of the state. This recalling of the high-reward experiences helps to increase sample efficiency and thus expedites the overall learning process (Blundell et al. 2016)
Pritzel al. 2017 Lin al. 2018). Please see Appendix A for related works and further discussions.
At timestep f, let us define global state as 8t. When utilizing episodic control, instead of directly using St, researchers adopt a state embedding function fo(s) : S E to project states toward k-dimensional vector space. With this projection, representation of global state becomes Ty- - f (st). The episodic control memorizes H (f)(())), i.e. the highest return of given global state $t, in episodic buffer DE (Pritzel et al.) 2017 Lin al., 2018 Zheng al. 2021). Here, I, is used as a key to the highest return, H (2z);; as a key-value pair in Dg. The episodic control in Lin
et al. 2018) updates H (1xe) with the following rules.
H (xx)) - max (āt), Re(s at)), if if - mell2 < Rs(s,aa))
otherwise.
where R2 (s,, ae) is the return of a given (st, az): 6 is a threshold value of state-embedding difference; and Rt (&((e) (+) is vz = fo(s))'s )ss nearest neighbor in Dg. If there is no similar projected state i such that || - < 5 in the memory, then Hxx:) keeps the current (se, at). Leveraging the episodic memory, EMC (Zheng et al. 2021) presents the one-step TD memory target QEa(fo(s) at)
ax) re(St, ax) + (fo(844))
(2)
Then, the loss function _ for training can be expressed the weighted sum of one-step TD error and one-step TD memory error, i.e., Monte Carlo (MC) inference error, based on Q (f))), at).
L FC
(y(s, a) - Qtoe(s, a; 8)) + X(Qrc(fo(s), a) - Qioc a;)))?
(3)
where y(8, a) is one-step TD target; Qtot is the joint Q-value function parameterized by 0; and X is scale factor.
Problem of the conventional episodic control with random projection Random projection is useful for dimensionality reduction as preserves distance relationships, as demonstrated by the Baanson-Lindenstranl
lemma (Dasgupta & Gupta 2003). However, a random projection adopted for fs(s) hardly has semantic meaning its embedding 3t, as it puts random weights on the state features without considering the patterns of determining the state returns. Additionally, when recalling the memory from D g. the projected state x, can abruptly change even with a small change of because the embedding is not being regulated by the return. This results in sparse selection of semantically similar memories, i.e. similar states with similar or better rewards. As a result, conventional episodic control using random projection only recalls identical states and relies on its own Monte-Carlo (MC) return to regulate the one-step TD target inference, limiting exploration of nearby states the embedding space.
The problem intensifies when the high-return states in the early training phase are indeed local optima. In such cases, the naive utilization of episodic control is prone converge local minima. As result, for the super hard tasks of SMAC, EMC (Zheng et al. (02T)) had to decrease the magnitude of this regularization to almost zero, i.e., not considering episodic memories anymore.
3

---

Published as a conference paper at ICLR 2024
3
METHODOLOGY
This section introduces Efficient episodic Memory Utilization (EMU) (Figure D explaining how construct (1) semantic memory embeddings to better utilize the episodic memory, which enables memory recall of similar, more promising states. To further improve memory utiliza- tion, an alternative the conventional episodic control, we propose (2) episodic incentive that selectively encourages desirable transitions while preventing local convergence towards undesirable trajectories.
We begin by
Environment
(a) Standard Value Factorization Framework
ladi
loffer. Iruli
(T, a'a, s.T) Replay Butfer
Controller Qi(:: 9)
Mixing
Mixing Network
Qtot /(Qu. 2z. 2ar Qu: 9)
Gradients
ss)(
PC
fe
f.)
(-)
P
(b) Eplsodic Butfer Dr-
(c) State Embedding Structure
(d) Eplsodic Incentive eneration
Figure 1: Overview of EMU framework.
3.1 SEMANTIC MEMORY EMBEDDING
Episodic Memory Construction To address the problems of a random projection adopted in episodic control, propose trainable embedding function fs(s) to learn the state embedding patterns affected by the highest return. The problem of a learnable embedding network So is that the match between H ([f((s)) and 8, breaks whenever f is updated. Hence, save the global state 8, as well as pair of H, and 2; in Dg, so that we can update x fs(s) whenever fo is updated. In addition, we store the desirability & of st according to Definition Appendix E. illustrates the details of memory construction proposed by this paper.
Learning framework for State Embedding When training (st), it is critical extract important features of a global state that affect its value, i.e., the highest return. Thus, we additionally adopt a decoder structure H, = (xx) to predict the highest return H, of 8t. We call this embedding function as EmbNet, and its learning objective of So and f can be written as
L(b, vb) (H, f (e)(s))))))
When constructing the embedding space, we found that an additional consideration of reconstruction of state conditioned timestep & improves the quality of feature extraction and constitutes smoother embedding space. To this end, we develop the deterministic conditional autoencoder (dCAE), and the corresponding loss function can be expressed
C( +b) = (Hl f (fo(ss)t)(t)) + Arcoml1s: (f (f
(5)
where JH predicts the highest return; f reconstructs 81; Aromn is scale factor. Here, I and f share the lower part of networks as illustrated in Figure([e) Appendix C.T presents the details of network structure of fo and fv. and Algorithm in Appendix C.I presents the learning framework for fo and fo. This training is conducted periodically in parallel to the RL policy learning on Qtoe 0).
p.epeesents
Figure illlustraaes the result of (-SNE (Van der Maaten & Hinton 2008) of 50K samples of æ E DE __________ 52 task of SMAC. Unlike supervised learning with label data, there is label for each Fz.. Thus, mark 2, with its pair of the highest return H,. Compared to a random projection in Figure 2(a), D, via fa is well-clustered, according to the similarity of the embedded state and its return. "This clustering of x, enables us safely select
out of IM memory data training for 3s_v____________________________
4

---

Published as a conference paper at ICLR 2024
(a) Random Projection
(b) EmbNet
(c) dCAE
Figure 2: t-SNE of sampled embedding x E Dg. Colors from red to purple (rainbow) represent from low return high return.
episodic memories around the key state 8t. which constitutes efficient memory utilization. This memory utilization expedites learning speed well encourages exploration state & near 8t. Appendix illustrates how to determine 6 of Eq. i memory-efficiet way.
promising
3.2 EPISODIC INCENTIVE
With the learnable memory embedding for an efficient memory recall, how to use the selected memories still remains a challenge because naive utilization of episodic memory is prone to converge on local minima. To solve this issue, we propose new reward structure called episodic incentive r" by leveraging the desirability of states in Dg- Before deriving the episodic incentive r", we first need to understand the characteristics of episodic control. In this section, we denote the joint Q-function (: 0) simply as Qu for conciseness.
Theorem 1. Given transition (s, a.rrss) and (x'), let Lg be the Q-learning loss with additional transition reward, i.e., Lg == (y(s, a) + F. (s. a, s') - Qtor a; 0))² where r EC (s, a, s') X(r(s, a) (H(() Qo a)), then Vol. V.
EC (Proof in Appendix B.
As Theorem I suggests, the additional transition reward EC (s, aaa.ss), s'). However, ,BC a, s') accompanies a risk of local convergence as discussed in Section 2.3 Therefore, instead of applying F a, s'),, we propose the episodic incentive rP == r)((s) that provides an additional reward for the desirable transition (s, a, s"), such that {(s') = 1. Here, rj(s') estimates 17" (s'), which represents the difference between the true value (s') of 8' and the predicted value via target network max, Qe- (s', a'), defined as
generate the same gradient signal as the episodic control by leveraging
n²(s)) = VV))=''(ss)) maxQo-
(6)
a'
Note that we do not know V*(s') and subsequently 17" (s'). To accurately estimate (x"( with i(s'), the expected value considering the current policy mo as ris') == Er, (n(')) where [0, Tiaax(ss (s')] for s' P(s'|s, (s, ~ πH). Here, (5") can be reasonably approximated by using (ff()s))) in DE. Then, with the count-based estimation fi(s), episodic incentive re can be expressed as
Ne(8')
Ne(s')
p >m(ss' = 7B. [n(s')] -
(Naul(s") (s)
Neau(s') (H(ff(s') maxQu (s',')),
where Noau(s') is the number of visits on E' NN(fo(s') DE; and Ng is the number of desirable transition from L'. Here, NN(-) represents a function for selecting the nearest neighbor. From Theorem [] the loss function adopting episodic control with alternative transition reward rP instead of FC can be expressed as
L = (r(s, a) + 1 Timaax (s', a') Qo(s,
(8)
Then, the gradient signal of the one-step TD inference loss VoLi with the episodic reward r" ri(s') can be written as
VeLi -V==-2VV0(s. +r)) - + Ng(s') Neaul (s') Tmax (s)),.
(9)
5

---

Published as a conference paper at ICLR 2024
where AETD r(s, ())s,) ymax, Q0 (s', Qs a) is one-step inference TD error. Here, the gradient signal VoLş with the proposed episodic reward y" can accurately estimate the optimal gradient signal follows.
Theorem 2. Let V -VVoo20 - (AETp) be the optimal gradient signal with the true one step TD error AETD a() 7V - Qo(s, a). Then, the gradient signal V VoLg with the episodic incentive r" converges to the optinal gradient signal as the policy converges to the optimal policy my. i.e.,
VOL. + VoLş as me + z.. (Proof Appendix B.2
Theorem Dllso implics that there ex- ists certain bias in V. as de- scribed in Appendix B.2 Besides the
3
property of convergence to the opti- mal gradient signal presented in The- orem 2] the episodic incentive has the following additional characteris- ties. (1) The episodic incentive is only applied to the desirable transi- tion. We can simply see that rP = yn >/E,,, [n] MMma /e/ Neall and if {(s') = then Ne = 0,
2
2000
(a) 3s5z
(b) MMM2
Figure 3: Episodic incentive. Test trajectories plotted on the embedded space with sampled memories in DE. denoted with dotted markers. Star markers and numbers represent the desirability of state and timestep in the episode, respectively. Color represents the same semantics as Figure
vielding rx + O. Subsequently, (2) there is no need to adjust a scale fac- tor by the task complexity. (3) The episodic incentive can reduce the risk of overestimation by considering the expected value of E [n]- Instead of considering the optimistic Tmaax the count-based estimation r ア = yYE.. [n] can consider the randomness the policy πи. Figure 3 illustrates how the episodic incentive works with the desirability stored in DE constructed by Algorithm presented in Appendix I In Figure 3 as we intended, high-value states (at small timesteps) are clustered close to the purple zone, while low-value states (at large timesteps) are located in the red zone.
3.3 OVERALL LEARNING OBJECTIVE
To construct the joint Q-function Qiot from individual Q; of the agent i. any form of mixer can be used. this paper, we mainly adopt the mixer presented QPLEX (Wang al. 2020b) similar to Zheng et al. (2021). which guarantees the complete Indiviuual-Global-
(IGM) condition (Son al. 2019: Wang al.) (020b). Considering any intrinsic reward re encouraging an exploration (Zheng al. 202 |) or diversity (Chenghao et al. (021), the final loss function for the action policy
learning from Eq. S can extended as
C3 (r(s,a) + r" + Bar + ymaxa Qtor(s' a'','':''''' a':0-> Qroe(s, a; 0))?
(10)
where B. is a scale factor. Note that the episodic incentive rD can be used in conjunction with any form of intrinsic reward yf being properly annealed throughout the training. Again, 0 denotes the parameters of networks related to action policy Q; and the corresponding mixer network to generate Qtot- For the action selection via Q, we adopt a GRU to encode a local oectionobseerattin history presented in 2. similar to (Sunchag et al. (007)) Rashid et al. (2018); Wang et al. (2020b); but in Eq. 10, we denote equations with s instead of for the coherence with derivation the previous section. Appendix E.2 presents the overall training algorithm.
EXPERIMENTS
In this part, we have formulated our experiments with the intention of addressing the following inquiries denoted as Q1-3.
Q1. How does EMU compare the state-of-the-art MARL frameworks?
Q2. How does the proposed state embedding change the embedding space and improve the performance?
Q3. How does the episodic incentive improve performance?

---

Published as a conference paper at ICLR 2024
We conduct experiments complex multi-agent tasks such SMAC (Samvelyan et al. 2019) and GRF (Kurach et al. 2020). The experiments compare EMU against EMC adopting episodic control (Zeng al. (021)) Also, include notable baselines, such value-based MARL methods QMIX (Rashid et al., 2018), QPLEX (Wang et al. 2020b), CDS encouraging individual diversity (Chenghao ee 2021). Particularly, we emphasize that EMU can be combined with any MARI. framework, so we present two versions of EMU implemented on original QPLEX and CDS, denoted as EMU (QPLEX) and EMU (CDS), respectively. Appendix Appendix[[provides further details of experiment settings and implementations, and Appendix D.12 provides the applicability of EMU to single-agent tasks, including pixel-based high- dimeeniooni tasks.
4.1
Q1. COMPARATIVE EVALUATION ON STARCRAFT II (SMAC)
EMU (QPLEX)
EMU (CDS)
QMIX
EMC
CDS
QPLEX
1c3s52
100
100
100
Test Vin [76]
0
0-
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
T (mi)
T (mil)
T (mil)
MMM2
6h _i_____ee _z_
355z vs, 356z
100
100
100
Test Win 1%5
50-
50
50
0-
1.0
2.0
1.0
2.0
1.0
2.0
3.0
T (mil)
T [mil)
T(mil)
Figure 4: Performance comparison of EMU against baseline algorithms on three easy and hard SMAC maps: 1c3s5z, 3s_vs__ ___,___,,_______________________ and 5m_vs_ 6m, and three super hard SMAC maps: MMM2, vs
6h_vs_ [h__v__2_ _2__2,___________ and 3s52_ vs__3622
Figure fillustrates the overall performance of EMU on various SMAC maps. The map categorization regarding the level of difficulty follows the practice of Samvellynn et al. (011).. Thanks to the efficient memory utilization and episodic incentive, both EMÜ (QPLEX) and EMU show significant performance improvement compared to their original methodologies. Especially, in super hard SMAC maps, the proposed method markedly expedites convergence on optimal policy.
4.2 Q1. COMPARATIVE EVALUATION ON GOOGLE RESEARCH FOOTBALL (GRF)
Here, we conduct experiments on GRF to further compare the performance of EMU with other baseline algorithms. In our GRF task, CDS and EMU (CDS) do not utilize the agent's index on observation as they contain the prediction networks while other baselines (QMIX, EMC, QPLEX) use information of the agent's identity in observations. In addition, algorithm, such as prioritized experience replay (Schaul et al., 2013), for all basclines and our method, to expedite learning efficiency. From the experiments, adopting EMU achieves significant performance improvement, and EMU quickly finds the winning or scoring policy at the early learning phase by utilizing semantically similar memory.
do not utilize any additional
EMU (QPLEX)
EMU (CDS)
QMIX
EMC
CDS
OPLEX
3. 3_v5_1WK
CA easy
CA_ hard
100
100
100
Test Win (S]]
50-
50-
50-
25
25
25
0.0
1.0
2.0
3.0
0.0
1.0
2.0
3.0
4.0
0.0
2.0
4.0
6.0
8.0
T
(ml)
T (mil)
T
(mil
Figure 5: Performance comparison of EMU against baseline algorithms on Google Research Football.
7

---

Published as a conference paper at ICLR 2024
4.3 Q2. PARAMETRIC AND ABLATION STUDY
In this section, we examine how the key hyperparameter 5 and the choice of design for Sa affect the performance. To compare the learning quality and performance more quantitatively, we propose new performance index called overall win-rate, Pa. The purpose of fay is to consider both training efficiency (speed) and quality (win-rate) for different seed cases (see Appendix [.] for details). We conduct experiments on selected SMAC maps to measure fue according to 5 and design choice for fo such as (1) random projection, (2) EmbNet with Eq. 4 and (3) dCAE with Eq.
08
100
100
To (N)
-004
204
Projction Enbrict
Rancorn Projecoo
Revirn Paceation
Errtket
Erdhal
SCAC
CEAC
OCAS
6CAF
68
0
-
1:0
1o98
100
1og0
(a) 3s_ve_52
(b) 3m_v__66________________________
(a) 35 52
(b) 5m vs 6m
Figure 6: Pn according to ō and various de- sign choices for So on SMAC maps.
Figure 7: Final win-rate according to d and various design choices for fg on SMAC maps.
Figure 6 and Figure 7 show The values and test win-rate at the end of training time according to different 6, presented in log-scale. To see the effect of design choice for fo distinctly, we conduct experiments with the conventional episodic control. More data of Pa is presented in Tables|aande 5 Appendix D.2] (D22] Figure Dillustraae that dCAE structure shows the best training efficiency throughout various 8 while achieving the optimal policy as other design choices as presented in Figure] 7
Interestingly, dCAE structure works well with a wider range of 5 than EmbNet. We conjecture that EmbNet can select very different states as exploration if those states have similar return H during training. This excessive memory recall adversely affects learning and fails to find an optimal policy as a result. See Appendix D.2 for detailed analysis and Appendix D.8 for an ablation study on the loss function of dCAE.
PSC ESUYOPEER
EHU-COS S:1
COUL 3-1
DVU rOPECK 6el LONEEC Eol
Sul
(a) CA_ hard (GRF)
(b) 6h vs _s_______________ 82 (SMAC)
Figure 8: Effect of varying on complex MARL tasks.
Even though a wide range of 6 works well as in Figures [6] and 7 choosing a proper value of 8 in more difficult MARL tasks significantly improves the overall learning performance. Figure g shows the learning curve of EMU according to 61 = 1.3e -7 82 = 1.3e 8g = 1.3e and 8₂ = 1.3e In super hard MARL tasks such 6h vs 8z in SMAC and CA hard in GRF, 63 shows the best 6b_vs___________________________
performance compared to other values. This is consistent with the value suggested in Appendix F where 8 is determined in memory-effciiett way. Further parametric study ö and Arcon are presented in Appendix [D. and D.6 respectively.
4.4
Q3. FURTHER ABLATION STUDY
In this section, we carry out further ablation studies to see the effect of episodic incentive rP presented in Section 3.2 From EMU (QPLEX) and EMU (CDS), ablate the episodic incentive and denote them with (No-EI). We additionally ablate embedding network fo from EMU and denote them with (No-SE). In addition, we ablate both parts, yielding EMC (OPLEX-original) and CDS (QPLEX- original). We evaluate the performance of each model on super hard SMAC maps. Additional ablation studies on GRF maps are presented in Appendix [D.] Note that EMC (QPLEX-original) utilizes the conventional episodic control presented in Zheng et al. (2021).
Figure 2 illustrates that the episodic incentive largely affects learning performance. Especially, EMU (QPLEX-No-ED) and EMU (CDS-No-EI) utilizing the conventional episodic control show large performance variation according to different seeds. This demonstrates that a naive utilization of episodic control could be detrimental learning optimal policy. On the other hand, the episodic incentive selectively encourages transition considering desirability and thus prevents such a local convergence. Appendix [19 and D.10 present an additional ablation study on semantic embedding
8

---

Published as a conference paper at ICLR 2024
(OFLEX
(OPLEX)
8000
80.0
60.0
[OPLEX
00.0
EMG aanggrr-n
Tee 750
ENC (UPL:S-anngea)
Toxt Won (50
(CPLLEX-nggna
400
40.0
40.0
260
20.0-
08
2,0
0.
0.0
3.0
0.0
Tink)
T(0kk)
T(re)
vz (a) _h_ va_ _ SMAC
(b)
_s_3362 SMAC
(c) 3s52_ve 1s_3s62 SMAC
Figure Ablation studies on episodic incentive via complex MARL tasks.
and r", respectively. In addition, Appendix D.II presents a comparison with an alternative incentive (Henaff et al. 2022) presented in single-agent setting.
4.5 QUALITATIVE ANALYSIS AND VISUALIZZTION
In this section, we conduct analysis with visualization to check how the desirability & is memorized in DE and whether conveys correct information. Figure [0] illustrates two test scenarios with different seeds, and each snapshot is denoted with corresponding timestep. In Figure the trajectory of each episode projected onto the embedded space DE.
In Figure 10 case (a) successfully de- feated all enemies, whereas case (b) lost the engagement. Both cases went through a similar, desirable trajectory at the beginning. For example, until t 10 agents in both cases focused on killing one enemy and kept all ally agents alive at the same time. How- ever, at / 12, case (b) lost one agent, and two trajectories of case (a) and (b) in embedded space began to bifur- cate. Case (b) still had a chance to win around f = 14 ~ 16. However, the states became undesirable (denoted without star marker) after losing three ally agents around t == 20, and case (b) lost the battle result. These sequences and characteristics of trajectories well captured by desirability Ç in Dr as illustrated in Figure
L-20
(a) Desirable trajectory 5m_vs 6m SMAC map
Iin
1-12
(b) Undesirable trajectory on 5m_vs_s________ 6m SMAC map
Figure 10: Visualization of test episodes.
Furthermore, the desirable state denoted with
1.25
Ç == 1 encourages exploration around though
0.76
1.50
0.75
1.00
it not directly retrieved during batch sampling. This occurs through the propagation of its desir- ability to states currently distinguished as unde- sirable during memory construction, using Algo- rithm 2 in Appendix E.I Consequently, when the state's is desirability is precisely memorized in Dg. it can encourage desirable transitions through the episodic incentive r!.
0.25
-0.25
-0.50
-0.50
-0.5
-0.75
5
5
Desirable trajectory
(b) Undesirable trajectory
Figure 11: Test trajectories on embedded space of
DE.
CONCLUSION
This paper presents EMU, a new framework to efficiently utilize episodic memory for cooperative MARL. EMU introduces major components: episodic incentive utilizing desirability of state. Semantic memory embedding allows us to safely utilize similar memory in wide area, expediting learning via exploratory memory recall. The proposed episodic incentive selectively encourages desirable transitions and reduces the risk of local convergence by leveraging the desirability of the state. As result, there is no need for manual hyperparameter tuning according to the complexity of tasks, unlike conventional episodic control. Experiments and ablation studies validate the effectiveness of each component EMU.
trainable semantic embedding and 2) an

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This research was supported by Al Technology Development for Commonsense Extraction, Rea- soning, and Inference from Heterogeneous Data(IITP) funded by the Ministry Science and ICT(2022-0-00077..
REFERENCES
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z. Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. Model-free episodic control. arXiv preprint 16Xivv1606.0444 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint 1rXiv:1810.12899 2018.
Li Chenghao, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. Celebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:3991-40000 2021.
Nikolay Dandanov, Hussein Al-Shatri, Anja Klein, and Vladimir Poulkov. Dynamic self-optimization of the antenna tilt for best trade-off between coverage and capacity in mobile networks. Wireless Personal Communications, 92(1):251 2017.
Sanjoy Dasgupta and Anupam Gupta. An elementary proof of a theorem of johnson and lindenstrauss. Random Structures & Algorithms, 22(1):60-66 2003.
Marc-André Dittrich and Silas Fohlmeister. Cooperative aulti-agent system for production control using reinforcement learning. CIRP Annals, 69(1 (1):3889-332, 2020.
Yali Du, Lei Han, Meng Fang, Liu, Tianhong Dai, and Dacheng Tao. Liir: Learning individual intrinsic reward in multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International conference autonomous agents and multiagent systems, pp. 66-83. Springer, 2017.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 aaai fall symposium series, 2015.
Mikael Henaff, Roberta Raileanu, Minqi Jiang, and Tim Rocktäschel. Exploration via elliptical episodic bonuses. Advances in Neural Information Processing Systems, 35:37631-37646, 2022.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances neural information processing systems, 29, 2016.
Hao Hu, Jianing Ye, Guangxiang Zhu, Zhizhou Ren, and Chongjie Zhang. Generalizable episodic memory for deep reinforcement learning. International conference machine learning, 2021.
10

---

Published as a conference paper at ICLR 2024
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International conference machine learning, pp. 9040-3049. PMLR, 2019.
Hyoungscok Kim, Jackyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. Exploration with mutual information. arXiv preprint arXiv: 10:::810000010 176, 2018.
Emi:
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprini arXiv: 1312.61 14, 2013.
Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajke, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. In Proceedings the AAAI Conference Artificial Intelligence, volume 34, pp. 4501-4510, 2020.
Lei Le, Andrew Patterson, and Martha White. Supervised autoencoders: Improving generalization performance with unsupervised regularizers. Advances in neural information processing systems, 31, 2018.
Máté Lengyel and Peter Dayan. Hippocampal contributions to control: the third way. Advances in neural information processing systems, 20, 2007.
Zichuan Lin, Tianqi Zhao, Guangwen Yang, and Lintao Zhang. Episodic memory deep q-networks.
arXiv preprint arXiv: 1805v:800.00000 007603, 2018.
Iou-Jen Liu, Unnat Jain, Raymond A Yeh, and Alexander Schwing. Cooperative exploration for multi-agent deep reinforcement learning. In International Conference on Machine Learning, pp. PMLR, 2021.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. Advances in Neural Information Processing Systems, 32, 2019.
David Henry Mguni, Taher Jafferjee, Jianhong Wang, Nicolas Perez-Nieves, Oliver Slumbers, Feifei Tong, Yang Li, Jiangcheng Zhu, Yaodong Yang, and Jun Wang. Ligs: Learnable intrinsic-reward generation selection for multi-agent learning. arXiv preprint arXiv:2112.02611 2021.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. Advances in neural information processing systems, 28, 2015.
Frans A Oliehoek and Christopher Amato. A concise introduction to decentralized POMDPs. Springer, 2016.
Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353, 2008
Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count- exploration with neural density models. In International conference on machine learning, pp. 2721-2730. PMLR, 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference machine learning, pp. 2778-2787. PMLR, 2017.
Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. In International Conference Machine Learning, pp. 2827-2832 PMLR, 2017.
Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel X Chang, et al. Habitat- matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv preprint arXiv:2 109, 08238, 2021.
11

---

Published as a conference paper at ICLR 2024
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shi- mon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International conference machine learning, pp. 4295-44992 44304. PMLR, 2018.
Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in neural information processing systems, 33:10199-10210, 2020.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint 190Xiv:902..44440 2019.
Tom Schaul, John Quan, Ioamnis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05953 2015.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances neural information processing systems, 28, 2015.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. International conference on machine learning, pp. 5887-5896. PMLR, 2019.
Bradly Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration reinforcement learning with deep predictive models. arXiv preprint arXiv: 2015.
Peter Sunchag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. delopppccopooiition networks for cooperative multi-agent learning, arXiv preprint arXiv: /0d.:05970609666 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurek, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. doi: 10.1 (09//ROS.2012..3866
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Binyu Wang, Zhe Liu, Qingbiao Li, and Amanda Prorok. Mobile robot path planning in dynamic environments through globally guided reinforcement learning. IEEE Roboties and Automation Letters, 5(4):6932-693) 2020a.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.0106,, 2020b.
Tonghan Wang, Jianhao Wang, Yi Wu, and Chongjie Zhang. Influence-based multi-agent exploration. arXiv preprint arXiv: 2019.
Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. In Proceedings of the International Conference Learning Representations (ICLR), 2021.
Marco A Wiering et al. Multi-agent reinforcement learning for traffic light control. In Machine Learning: Proceedings of the Seventeenth International Conference (ICML'2000) pp. 1151-1158, 2000.
Jiachen Yang, Igor Borovikov, and Hongyuan Zha. Hierarchical cooperative multi-agent reinforce- ment learning with skill discovery. arXiv preprint arXiv: 90999/23333 2019.
12

---

Published as a conference paper at ICLR 2024
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03939, 2020.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611-24624, 2022.
Lulu Zheng, Jiarui Chen, Jianhao Wang, Jiamin He, Yujing Hu, Yingfeng Chen, Changjie Fan, Yang Gao, and Chongjie Zhang. Episodic multi-agent reinforcement learning with curiosity-driven exploration. Advances Neurul Information Processing Systems, 34:3757-3769, 2021.
Guangxiang Zhu, Zichuan Lin, Guangwen Yang, and Chongjie Zhang. Episodic reinforcement learning with associative memory. International conference on learning representations, 2020.
13

---

Published as a conference paper at ICLR 2024
A RELATED WORKS
This section presents the related works regarding incentive generation for exploration, episodic control, and the characteristics of cooperative MARL.
A.1 INCENTIVE FOR MULTI-AGENT EXPLORATION
Balancing between exploration and exploitation in policy learning is a paramount issue in reinforce-
ment learning. To encourage exploration, modified count-based methods (Bellemare et al. 2016)
Ostrovski et al. al.| 2017: Tang et al. 2017). prediction error-based methods (Stadie et al.) 2015] Pathak
et al. 2017] Burda et al. 201 8 Kim et al., 2018). and information gain-based methods (Mohamed &
Jimenez Rezende 2013 Houthooft et al. 2016) have been proposed for a single agent reinforcement learning. In most cases, an incentive for exploration is introduced
additional reward to TD target in Q-learning; or such an incentive is added as a regularizer for overall loss functions. Recently, various aforementioned methods to encourage exploration have been adopted to the multi-agent setting (Mahajan et al., [019]] Wang et al. 2019| Jaques et al., 2019: Mguni et al. 2021) and have shown their effectiveness. MAVEN (Mahajan et al., 2019) introduces regularizer maximizing the mutual information between trajectories and latent variables to learn a diverse set of behaviors. LII (Du al. 2019) learns a parameterized individual intrinsic reward function by maximizing centralized critic. CDS (Chenghao et al., (222]) proposes a novel information-theww -teooretiall objective to maximize the mutual information between agents' identities and trajectories encourage diverse individualized behaviors. EMC (Zheng et al., 2021) proposes
exploration by predicting individual Q-values. This individual based Q-value prediction can capture the influence among agents well as the novelty of states.
A.2 EPISODIC CONTROL
Episodic control Lengyel Dayan 2007 was well adopted on model-free setting (Blundell et al. 2016) by storing the highest return of a given state, to efficiently estimate its values or Q-values. Given that the sample generation is often limited by simulation executions or real- world observations, its sample efficiency helps to find an accurate estimation of Q-value (Blundell et al. 2016 Pritzel ct al. 2017 Lin al., 2018). NEC (Pritzel et al. 2017) uses a differentiable neural dictionary as an episodic memory to estimate the action value by the weighted sum of the values in the memory. EMDQN (Lin et al. 2018) utilizes a fixed random matrix to generate a state representation, which is used as a key to link between the state representation and the highest return of the state in the episodic memory. ERLAM (Zhu et al., (020) learns associative memories by building a graphical representation of states in memory, and GEM (Hu et al. (021) develops state-action values of episodic memory in a generalizable manner. Recently EMC (Zheng et al. [021) extended the approach of EMDQN to a deep MARL with curiosity-driven exploration incentives. EMC utilizes episodic memory to regularize policy learning and shows performance improvement in cooperative MARL tasks. However, EMC requires hyperparameter tuning determine the level of importance of the one-step TD memory-based target during training, according to the difficulties of tasks. In this paper, we interpret this regularization as an additional transition reward. Then, we present a novel form of reward, called episodic incentive, to selectively encourage the transition toward desired states, i.e., states toward a common goal in cooperative multi-agent tasks.
A.3 COOPERATIVE MULTI-AGENT REINFORCEMENT LEARNING (MARL) TASK
In general, there is a common goal in cooperative MARL tasks, which guarantees the maximum return that can be obtained from the environment. Thus, there could be many local optima with high returns but not the maximum, which means the agents failed to achieve the common goal in the end. In other words, there is a distinct difference between the objective of cooperative MARL tasks and that of a single-agent task, which aims to maximize the return as much as possible without any boundary determining success or failure. Our desirability definition presented in Definition I in MARL setting becomes well justified from this view. Under this characteristic of MARL tasks, learning optimal policy often takes a long time and even fails, yielding a local convergence. EMU was designed to alleviate these issues in MARI..
14

---

Published as a conference paper at ICLR 2024
B MATHEMATICAL PROOF
In this section, we present the omitted proofs of Theorem Jand Theorem as follows.
B. PROOF OF THEOREM
I
Proof. The loss function of a conventional episodic control, L can be expressed as the weighted sum of one-step inference TD enror ASTD = r(s, a) + ymaxx Qa (8', a') - Q0(8, and MC inference error A = QEC(s, a) Qo(s,a).
LB = (r(s, a) + 7maxxx= (5', Qo(s, a))² X(QEa a) Q)(+,a))² (1I)
a' where QEC(s, a) = r(s,, a) + (s')) and Qo- is the target network parameterized by 0_. Then, the gradient of Lg can be derived as
VoLg - -V (0 a) +m (s',a') - Qo(s, XQQ Q0(s, a))]
())Qo(s.a) AAEEC).
(12)
Now, we consider additional reward EC for the transition conventional Q-learning objective, the modified loss function Lg can be expressed as
Lg (r(s, a) + EC (s, a, s') + ymaxQe- (s', a') Qo(s,
(13)
Then, the gradient of Le presented in Eq. 13 computed
VgLg V al)ssa)((eT))) + BC).
(14)
Comparing Eq. 12 and Eq. 14 if we set the additional transition reward , (8,
A(r(s, a) + 7 (s a)), then VoLo = VoL. holds.
B.2 PROOF OF THEOREM 2
Proof. From Eq. 7 the value of ij(') can be expressed as
n(s') E.. (n(ss))))))))) Ne(s')
(s) (H(fo(s') maxQo-(s',
(15)
When the joint actions from the current time follow the optimal policy, a ~ ny. the cumulative reward from s' converges to (=(s'), i.e., H(fo(s') + Vs').. Then, every recall of /' NN(so(s') DE guarantees the desirable transition, i.e., E(s') = 1, where NN(-) represents function for Ne(s')
selecting the nearest neighbor. a result, as (sal((s') 0o, Neau((") Ne(s')
1, yielding rjs') ~ Neau(e") (H(fa(s')) - max, (0-(('' VV((') 10xx-maaaaaa Qo (s', a'). Then, the gradient signal
with the episodic incentive L becomes
V I.
2V AETD + r?]
-V,Qo(s a) AETD + + 78((s))
2VeQQ((s a) AETD + (V (V* (s') maxx0 a'))
-VVoQQ 2V + (s' ao(s. +2( maxQ (s', a'))
--VV a)rr (V' Qs(s.a)
VoLi.
(16)
which completes the proof.
In addition, when 1 (s', a') accurately estimates (s'),, the original TD-target is preserved as the episodic incentive becomes zeto, i.c., rP 0. Then with the properly annealed intrinsic reward re, the learning objective presented in Eq. 10 degenerates to the original Bellman optimality equation Sutton & Barto 2018). On the other hand, even though the assumption of H(s') - V (s') yields AEEG + A L EC has an additional bias ASTD due to weighted sum structure presented in
EC can converge to VoLğ only when max, Qp- (s', a') - V*(s') and + at the same time.
Eq. 3 Thus, VoL
15

---

Published as a conference paper at ICLR 2024
C IMPLEMENTATION AND EXPERIMENT DETAILS
C.I DETAILS OF IMPLEMENTATION
Encoder and Decoder Structure
As illustrated in Figure [ have an encoder and decoder structure. For encoder f evaluate two types of structure, EmbNet and dCAE. For EmbNet with the learning objective presented Eq. 4 two fully connected layers with 64-dimensional hidden state are used with ReLU activation function between them, followed by layer normalization block at the head. On the other hand, for dCAF with the learning objective presented in Eq. 5 we utilize deeper encoder structure which contains three fully connected layers with ReLU activation function. In addition, dCAE takes a timestep & as an input as well as a global state st. We set episodic latent dimension dim (a) = 4 as Zheng al. 2021)
Laver
Rel.I
S
X
MHU
Rel.U U
RaLu
())
(a) EmbNet
(b) dCAE
Figure 12: Illustration of network structures.
For decoder fu, both EmbNet and dCAE utilize three-fully connected layer with ReLU activation functions. Differences are that EmbNet takes only sa as input and utilizes the 128-dimensional hidden state while dCAF takes x, and as inputs and adopts the 64-dimensional hidden state. As illustrated in Figure C)). to reconstruct global state st, dCAE has two separate heads while sharing lower parts of networks; f to reconstruct s; and f predict the return of s;, denoted Hç. Figure 12 illustraases network structures of EmbNet and dCAE. The concept of supervised VAE similar to EMU can be found in (Le et al. 2018).
The reason behind avoiding probabilistic autoencoders such variational autoencoder (VAE) (Kingma & Welling 2013 Sohnn et al.) 2015) is that the stochastic embedding and the prior distribu- tion could have an adverse impact on preserving a pair 2, and H, for given a st. In particular, with stochastic embedding, fixed can generate diverse It. As result, it breaks the pair of x, and H; for given St, which makes difficult select valid memory from DE.
For training, we periodically update fe and fø with update interval of femb in parallel to MARL training. At each training phase, we use Mem' samples of the current capacity of Dz. whose maximum capacity is million (1M), and batch size of Memb is used for each training step. After updating fe, every 2 E 'Dg needs to be updated with updated fa- Algorithm T shows details of learning framework for fo and fg. Details of the training procedure for fo and along with MARL training are presented in Appendix E.2
Other Network Structure and Hyperparameters
For a mixer structure, we adopt QPLEX (Wang et al. (020b)) in both EMU (QPLEX) and EMU (CDS) and follow the same hyperparameter settings used in their source codes. Common hyperparameters related to individual Q-network and MARL training are adopted by the default settings of PyMARL
(Samvelyan al. 2019).
16

---

Published as a conference paper at ICLR 2024
Algorithm 1 Training Algorithm for State Embedding
1: Parameter: learning rate a, number of training dataset N, batch size B
2: Sample Training dataset (s(+), (H) f(0) DE.
N
3: Initialize weights Φ, & 0
4: for i = to N B do
5:
Compute (x = f (ss5)) i8
j= 11++1
6: Predict return (H = / (x(0 4 (111))
7: Reconstruct state (g(
(z(
()(1-))B+
8: Compute Loss L(Φ,, ab) via Eq. 5
HC 9: Update Φ t Φ DΦ'
vb t v
ac
10:
end for
C.2 EXPERIMENT DETAILS
We utilize PyMARL (Samvelyan et al. 2019) to execute all of the baseline algorithms with their open-source codes, and the same hyperparameters are used for experiments they are presented either in uploaded codes or in their manuscripts.
For a general performance evaluation, we test our methods on various maps, which require a different level of coordination according to the map's 's difficulties. Win-rate is computed with 160 samples: 32 episodes for each training random seed, and 5 different random seeds unless denoted otherwise.
Both the mean and the variance of the performance presented for all the figures show their overall performance according to different seeds. Especially for a fair comparison, we set Picircles the number of training per sampled batch of 32 episodes during training, some of the baselines increase neireeee 2 as a default setting in their codes.
for all baselines since
For performance comparison with baseline methods, we use their codes with fine-tuned algorithm configuration for hyperparameter settings according to their codes and original paper if available. For experiments on SMAC, we use the version of starcraft py presented in RODE (Wang et al. 2021) adopting some modification for compatibility with QPLEX (Wang et al. 2020b). All SMAC experiments were conducted on StarCraft II version 4.10.0 in Linux environment.
For Google research football task, we use the environmental code provided by (Kurach et al. 2020). In the experiments, we consider three official scenarios such as academy_3 ___vs___________________ with keeper (3 vs IWK), academy saaddmmy_couterrttaaee easy (CA @e__ess), and academy academy_ccontercnn cownterattack hard (CA _ardd.
In addition, for controlling in Eq. 10 the same hyperparameters related to curiosity- based (Zheng et al. 2021 or diversity-based exploration Chenghao et al. (2021) are adopted for EMU (QPLEX) and EMU (CDS) as well as for baselines EMC and CDS. After further experiments, we found that the curiosity- bbsed rf from (Zheng et al. 2021) adversely influenced super hard SMAC task, with the exception of corridor scenario. Furthermore, the diversity- -bseed exploration from Chenghao both easy and hard SMAC maps. Thus, to exclude the use of r for EMU (QPLEX) on the super hard SMAC task and for EMU (CDS) on the easy/hard SMAC maps. EMU set task-dependent ó values as presented in Table [ For other hyperparameters introduced by EMU, the same values presented in Table 8 are used throughout all tasks. For EMU (QPLEX) in corridor, 1.3e 5 is used instead of o 1.3e 3.
al. 2021) led to decrease in performance
decided
Table 1: desk-dependent hyperparameter of EMU.
Category
easy/hard SMAC maps
1.3e 5
super hard SMAC maps
1.3e
GRF
1.3e
17

---

Published as a conference paper at ICLR 2024
C.3 DETAILS OF MARL TASKS
In this section, we specify the dimension of the state space, the action space, the episodic length, and the reward of SMAC (Samvelyan et al. 2019) and GRF (Kurach et al. 2020).
In SMAC, the global state of each task of SMAC includes the information of the coordinates of all agents, and features of both allied and enemy units. The action space of each agent consists of the moving actions and attacking enemies, and thus increases according the number of enemies. The dimensions of the state and action space and the episodic length vary according to the tasks as shown T For reward structure, we used the shaped reward, i.e., the default reward settings of SMAC, for all scenarios. The reward is given when dealing damage to the enemies and get bonuses for winning the scenario. The reward is scaled so that the maximum cumulative reward, Rmaz. that can be obtained from the episode, becomes around 20 (Samvelyan et al. 2019)
Table 2: Dimension of the state space and the action space and the episodic length of SMAC
Task
Dimension of state space
Dimension of action space
Episodic length
lc3s5z
270
180
3s5z
216
14
150
3s_vs__
68
250
5m
6m
98
12
70
MMM2
322
18
180
6h_ 8z
140
14
150
3s5z_vs_3std
230
15
170
corridor
282
30
400
In GRF, the state of each task includes information on coordinates, ball possession, and the direction of players, etc. The dimension of the state space differs among the tasks as in Table 3 The action of each agent consists of moving directions, different ways kick the ball, sprinting, intercepting the ball and dribbling. The dimensions of the action spaces for each task are the same. Table summarizes the dimension of the action space and the episodic length. GRF, there can be two reward modes: one is "sparse reward" and the other is "dense reward." In sparse reward mode, agents get a positive reward +1 when scoring a goal and get when conceding one to the opponents. In dense reward mode, agents can get positive rewards when they approach opponent's 's goal, but the maximum cumulative reward is up to +1. In our experiments, we adopt sparse reward mode, and thus the maximum reward, Rmaz is +1 for GRF.
Table 3: Dimension of the state space and the action space and the episodic length of GRF
Task
Dimension of state space
Dimension of action space
Episodic length
3_ vss 1_
26
19
150
CA_easy
30
19
150
CA hard
34
19
150
C.4 INFRASTRUCTURE
Experiments for SMAC (Samvelyan et al. 2019) are mainly carried out on NVIDIA GeForce RTX 3090 GPU, and training for the longest experiment such as corridor via EMU (CDS) took less than 18 hours. Note that when training is conducted with Meirele = 2, it takes more than one and half times longer. Training encoder/decoder structure and updating D with updated fa together only took less than 2 seconds most in corridor task. As update fo and fo periodically with tembs the additional time required for a trainable embedder is certainly negligible compared to MARL training.
18

---

Published as a conference paper at ICLR 2024
D FURTHER EXPERIMENT RESULTS
D.1 NEW PERFORMANCE INDEX
In this section, we present the details of new performance index called overall win-rate, Pw- For example, let f (s) be the test win-rate at training time of ith seed run and p (f) represents the time integration of f (8) until t. Then, a normalized overall win-rate, pwu, can be expressed as
1 1 E L. fi)ss Mmax n2
(w) Pmax
M (t)
(17)
where Umax and Pw [0, 1].
Test Win []
(s)
f (s)
(t)
Figure 13: Illustration of (t).
Figure 13 illustrates the concept of time integration of win-rate, i.e., ph (t), to construct the overall win-rate, fww- By considering the integration of win-rate of each seed case, the performance variance can be considered, and thus Te shows the training efficiency (speed) as well as the training quality (win-rate).
D.2 ADDITIONAL EXPERIMENT RESULTS
present the summary of parametric studies å with respect to various choices of fo- To see the training efficiency and performance at the same time, Table and present the overall win-rate Au according training time. We conduct the experiments for different seed cases and at each test phase 32 samples were used to evaluate the win-rate [%]. Note that we discard the component of episodic incentive yP to see the performance variations according to å and types of fō more clearly.
In Section 4.3
Table 4: Awr according to 8 and design choice of embedding function hard SMAC map, 3s_vs_5z
Training time [mil]
0.69
1.37
2.00
random
EmbNet
dCAE
random
EmbNet
dCAE
random
EmbNet
dCAE
1.3e-7
0.033
0.051
0.075
0.245
0.279
0.343
0.413
0.443
0.514
1.3e-5
0.010
0.044
0.063
0.171
0.270
0.325
0.320
0.441
0.491
1.3e-3
0.034
0.043
0.078
0.226
0.270
0.357
0.381
0.439
0.525
1.3e-2
0.019
0.005
0.079
0.205
0.059
0.346
0.348
0.101
0.518
Table 5: Au according to and design choice of embedding function on hard SMAC map.
5m vs 6m.
Training time
0.69
1.37
2.00
[mil]
random
EmbNet
dCAE
random
EmbNet
dCAB
random
EmbNet
dCAA
1.3e-7
0.040
0.117
0.110
0.287
0.397
0.397
0.577
0.690
0.701
1.3e-5
0.064
0.107
0.131
0.334
0.402
0.436
0.634
0.714
0.749
1.3e-3
0.040
0.080
0.064
0.333
0.377
0.363
0.646
0.687
0.677
1.3e-2
0.038
0.000
0.048
0.288
0.001
0.332
0.584
0.001
0.643
19

---

Published as a conference paper at ICLR 2024
As Table 4 and 5 illustrate that dCAE structure for fø. which considers the reconstruction loss of global state s as in Eq. [S] shows the best training efficiency in most cases. For innvs__ 6m task with 5 = 1.3e EmbNet achieves the highest value among fg choices in terms of f.u but fails to find optimal policy at å = 1.3e unlike other design choices. This implies that the reconstruction loss of dCAE facilitates the construction of a smoother embedding space for Dg, enabling the retrieval of memories within broader range of 5 values from the key state. Figure 15 and I6 show the corresponding learning curves of each encoder structure for different ō values. A large à value results in higher performance variance than the cases with smaller 6, according to different seed cases.
This is because a high value of encourages exploratory memory recall. In other words, by adjusting 5, we can control the level of explo- ration since it controls whether to recall its own MC return or the highest value of other simi- lar states within 5. Thus, without constructing smoother embedding space as in dCAE, learn- ing with exploratory memory recall within large ó can converge to sub-optimality as illustrated by the case of EmbNet in Figure d))
horston Irqectionn
Eribhit
dDAF
05
1.0
Figure 14: Neaul of all memories in DE when
8 = 0.013 according to design choice for fo-
In Figure 14 which shows the averaged number of memory recall (Nontt) of all memories in DE. Neau of EmbNet significantly increases as training proceeds. On the other hand, dCAE was able to prevent this problem and recalled the proper memories in the early learning phase, achieving good training efficiency. Thus, embedding with dCAE can leverage a wide area of memory in Dg and becomes robust to hyperparameter d.
Maccaa
65AE
GSAE
6CA2
CSAE
Tim0)
1inE
1ime
(a) 8 1.3e
(b) 5 1.3e
(c) 8 1.3e
(d) 5 1.3e
Figure 15: Parametric studies for on 3s_vs_5t SMAC map.
X
R
4
Prietie
Reaie Prietie
Renire
6
19
15
23
60
65
10
15
20
50
85
10
15
20
50
85
10
15
20
(a) 8
1.3e -7
(b) 8 = 1.3e 5
(c) 8 = 1.3c -3
(d) 5 = 1.3e 2
Figure 16: Parametric studies for d on 5m vs 6m SMAC map.
20

---

Published as a conference paper at ICLR 2024
D.3 COMPARATIVE EVALUATION ON ADDITIONAL STARCRAFT II MAPS
Figure 17 presents a comparative evaluation of EMU with baseline algorithms on additional SMAC maps. Adopting EMU shows performance gain in various tasks.
EMU (QPLEX)
EMU (CDS)
QMIX
EMC
CDS
QPLEX
8m
3s5z
2s vs. _s_
corridor
100
100
100
100
Test Win [%6
50
50
0
0.0
0.5
1.5 T (mil)
1.0
2.0
0.0
0.5
1.0 (mil)
1.5
2.0
0.0
0.5
1.0 (mil)
1.5
2.0
0.0
1.0
2.0
3.0
4.0
T (mil)
Figure 17: Performance comparison of EMU against baseline algorithms on additional SMAC maps.
D.4 COMPARISON OF EMU WITH MAPPO ON SMAC
In this subsection, we compare the EMU with MAPPO YYu al. 2022 on selected SMAC maps. Figure [8] shows the performance in six SMAC maps: sc3552, 3s_vs
vs vs_52, 5m_ 6m. MMM2, 6h_vs_8 8z and 3s52_ __s 3362. Similar to the previous performance evaluation in Figure 4 Win-rate is computed with 160 samples: 32 episodes for each training random seed and 5 different random seeds. Also, for MAPPO, scenario-dependent hyperparameters are adopted from their original settings in the uploaded source code.
we can see that EMU performs better than MAPPO with an evident gap. Although after extensive training MAPPO showed comparable performance against off- algorithm in its original paper YYu et al. 2022). within the same training timestep used for our experiments, we found that MAPPO suffers from local convergence in super hard SMAC tasks such MMM2 and 3s52_vs__ _r____3__0 6z as shown in Figure 18] Only in Gh_vv_B MAPPO shows comparable performance to EMU (QPLEX) with higher performance variance across different seeds.
From Figure 18
(QPLEX)
(CDS)
MAPPO
1c3s5z
38
6m
100
100
100
Test Win 1%5]
50
50
50
0.0
1.5
0.0
1.5
0.0
1.5
T (mil)
T (mil)
T (mil)
3s52 vs_
100
100
100
Test Win (%6]
50
50
50
0.0
1.0
2.0
3.0
4.0
0.0
1.0
2.0
3.0
4.0
0.0
1.0
2.0
3.0
4.0
T
T
(mil)
T (mil)
Figure 18; Performance comparison with MAPPO on selected SMAC maps.
21

---

Published as a conference paper at ICLR 2024
D.5 ADDITIONAL PARAMETRIC STUDY
In this subsection, we conduct an additional parametric study to see the effect of key hyperpa- rameter 8. Unlike the previous parametric study on Appendix D.2 we adopt both dCAE em- bedding network for fa and episodic reward. For evaluation, we consider three GRF tasks such as academy_3_ _________ww_ S___wtt_keeeeeeeee (CA-easy), and seadleey_counterrtteeee haaaraarddaaddaa (CA-hard); and one super hard SMAC map such as 6h_ _2. For each task to evaluate EMU, four values, such as 61 = 1.3e 52 = 1.3e 63 1.3e and 64 1.3e-2 2, are considred. Here, to compute the win-rate, 160 samples (32 episodes for each training random seed and 5 different random seeds) are used for vs s_v___We 1WK and 6h v___v________ 82 while 100 samples (20 episodes for each training random seed and different random seeds) are used for CA-easy and CA-hare Note that CDS and EMU (CDS) utilize the same hyperparameters, and EMC and EMU (QPLEX) the same hyperparameters without a curiosity incentive presented in iZheng et al. (2021) as the model without it showed the better performance when utilizing episodic control.
(3_vs_1WB3) academy Secuddeyyccooeteeeeeeee www.oottetttttooooooooo rateertttacc ck _
CBS ENUICOO
EXC
OMU XSS
ENUISPEER
EIUIDESS- d:) EMUICOS..M
EMR1IC35L Sal
CSPIL PJ
T(00
7000
(a) 3_vs_____ 1WK (GRF)
(b) CA-easy (GRF)
(c) CA-hard (GRF)
(d) 6h_vs 8z (SMAC)
Figure 19: Parametric studies for on various GRF maps and super hard SMAC map.
In all cases, EMU with 63 = 1.3e-3 shows the best performance. The tasks considered here all complex multi-agent tasks, and thus adopting a proper value of benefits the overall performance and achieves the balance between exploration and exploitation by recalling the semantically similar memories from episodic memory. The optimal value of d3 is consistent with the determination logic on in a memory efficient way presented in Appendix F
D.6 ADDITIONAL PARAMETRIC STUDY ON Xyoon
Additionally, we conduct a parametric study for Aycon in Eq. For each task, EMU with five Areon values, such Areon, 0.01, Arcon,1 0.1, Avcon.2 0.5, Areon,3 = 1.0 and Avom, = 10, are evaluated. Here, to compute the win-rate of each case, 160 samples (32 episodes for each training random seed and different random seeds) are used. From Figure 20,
that broad range of
100
EMU (aaual EMU (hanal)
10
EML (Panc))
EMI (Pess)
EMI (Pers))
20-
20-
02
1.0
20
1.0
1.5
(a) 3s5z (SMAC)
(b) 33 VS_ ______200 (SMAC)
Figure 20: Parametric study for Arcon.
Aroon E {0.1, 0.0, 1.0} work well in general. However, with large Arcon as Arcon,4 = 10, we can observe that some performance degradation at the early learning phase in 3s5z task. This result is in line with the learning trends of Case I and Case 2 of 3s52 in Figure 23 which do not consider prediction loss and only take into account the reconstruction loss. Thus, considering both prediction loss and reconstruction loss as Case 4 in Eq. 51 proper Arcon is essential to optimize the overall learning performance.
22

---

Published as a conference paper at ICLR 2024
D.7 ADDITIONAL ABLATION STUDY IN GRF
(CDB)
EMU (CDS)
(CD8-No
(CBS-A (CDS (CPLEX-oigHe))
CDS 30PL EX-ontroa))
60-
Text
40-
20-
0.0
0.5
1.5
2.5
(mi))
(m00)
(a) 3_ VS
1s__vs___ 19K (GRF)
(b) CA CA-easy (GRF)
Figure 21: Ablation studies on episodic incentive on GRF tasks.
In this subsection, we conduct additional ablation studies via GRF tasks see the effect of episodic incentive. Again, EMU (CDS-No-EI) ablates episodic incentive from EMU (CDS) and utilizes the conventional episodic control presented in Eq. Bl instead. Again, EMU (CDS-No-SE)) ablates semantic embedding by dCAF and adopts random projection with episodic incentive r?. In both tasks, utilizing episodic memory with the proposed embedding function improves the overall performance compared to the original CDS algorithm. By adopting episodic incentives instead of conventional episodic control, EMU (CDS) achieves better learning efficiency and rapidly converges to optimal policies compared to EMU (CDS-No-EI).
D.8 ADDITIONAL ABLATION STUDY ON EMBEDDING LOss
In our case, the autoencoder uses the reconstruction loss to enforce the embedded representation x contain the full information of the original feature, s. We are adding (H, - # ( (s)()) to guide the embedded representation to be consistent Hr, well, which works regularizer the autoencoder. Therefore, H is used Eq. to predict the observed H, from DE as a part of the semantic regularization effort.
Because H, is different from f (xt), the effort of minimizing their difference becomes the regularizer creating a gradient signal to learn and Φ. The update of ó results the updated x influenced by the regularization. Note that update Φ through the backpropagation of vb.
The case of L(d, v- = || - (s)((()))) occurs when Arcon becomes relatively much higher than 1, which makes (H,- / (Jo(s.)))))) 1))) becomes ineffective. In other words, when Xrcon in Eq. becomes relatively much higher than 1, (H, - (fg(s((t)(t)) becomes ineffective.
The case L(O, v/ = (H; - (()()))))))) occurs when the scale factor Xreon becomes relatively much smaller than 1, which makes (H, f (s)))tt)) become dominant factor. We conduct ablation studies considering four cases as follows:
Case 1: L(o, +) || (f presented in Figure 22 a)
Case 2: I(d, eb) ||st { presented in Figure 22 b)
Case 3: L(d, vb) (He -
) (fo( presented in Figure 22 c)
Case 4: T(( (H, - (3o( Arcon ||s [0(((())0)))) i.e., Eq.
presented Figure 22
We visualize the result of t-SNE 50K samples x E DE out of 1M memory data trained by various loss functions: The task was 3s_vs_55 of SMAC as in Figure 2 and the training for all models proceeds for 1.5mil training steps. Case and Case 2 showed irregular return distribution across the embedding space. In those two cases, there was no consistent pattern in the reward distribution. Case 3 with only return prediction in the loss showed better patterns compared to Case and 2 but some features not clustered well. We suspect that the consistent state representation also contributes to the return prediction. Case 4 of our suggested loss showed the most regular pattern in the return distribution arranging the low-return states cluster and the states with desirable returns another
23

---

Published as a conference paper at ICLR 2024
20
8
R
-25
0
-20
-00
40
-75
(a) Loss (case 1)
(b) Loss (case 2)
(c) Loss (case 3)
(d) Loss (case 4)
Figure 22: t-SNE sampled embedding x E Dg trained by dCAE with various loss functions in 35 vs 52 SMAC map. Colors from red purple represent from low return high return.
cluster. In Figure 23 Case 4 shows the best performance in terms of both learning efficiency and terminal win-rate.
ZVU Joan
08
1.5
15
T199
3s5z (SMAC)
(b) 5z (SMAC)
Figure 23: Performance comparison various loss functions for dCAE.
24

---

Published as a conference paper at ICLR 2024
D.9 ADDITIONAL ABLATION STUDY ON SEMANTIC EMBEDDING
To further understand the role of semantic embedding, we conduct additional ablation studies and present them with the general performance of other baseline methods. Again, EMU (CDS-No-SE) ablates semantic embedding by dCAE and adopts random projection instead, along with episodic incentive rP.
EMU (OPLEX)
QMIX
CDS
QFLEX
EMU (OPLEX-NO-SE)
(CDS-No-SE)
EMU (CDS)
EMC
163652
36
5m. vs. 8.m
100
100
100
[%)
50
50
50
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
(mil)
(mll)
MMM2
8h vs. 8z
365z vs. 388z
100
100
Test (%)
50-
50
50
0.0
1.0
2.0
3.0
4.0
0.0
1.0
2.0
3.0
4.0
0.0
1.0
2.0
3.0
4.0
[mil)
(mil)
(mil)
Figure 24: Performance comparison of EMU against baseline algorithms on three easy and hard SMAC maps: 1 c3,5 2______000 __,, and 5m_vs_6nt and three super hard SMAC maps: MMM2, 6h vs 8z, and 3s52 3z
CDB
GPLEX
ENU (OPLEX
(CDS SE)
(CDS)
3. 1WK
100
CA 500y
50
80
Toe WN
80
20-
0.0
1.0
2.0
3.0
0.5
1.0
2.0
3.0
4.0
T(ml)
(m4])
Figure 25: Performance comparison of EMU against baseline algorithms Google Research Football.
For relatively easy tasks, EMU (QPLEX-No-SE) and EMU (CDS-No-SE) show comparable perfor- mance at first but they converge on sub-optimal policy most tasks. Especially, this characteristic is well observed in the case of EMU (CDS-No-SE) As large size of memories are stored in an episodic buffer training goes on, the probability of recalling similar memories increases. However, with random projection, semantically incoherent memories can be recalled and thus it can adversely affect the value estimation. We deem this the reason for the convergence on suboptimal policy in the case of EMU (No-SE). Thus we can conclude that recalling semantically coherent memory is an essential component of EMU.
25

---

Published as a conference paper at ICLR 2024
D.10 ADDITIONAL ABLATION ON pd
In Eq 10, we introduce rf as an additional reward which may encourage exploratory behavior or coordination. The reason we introduce rf is show that EMU can be used in conjunction with any form of incentive encouraging further exploration. Our method may not be strongly effective until some desired states are found, although it has exploratory behavior via the proposed semantic embeddings, controlled by 8. Until then, such incentives could be beneficial to find desired or goal states. Figures 27 show the ablation study of with and without r", and the contribution of r is limited compared to r".
(CR
DMU IONLDX DMC
DNU JOFLEX,
DNC
Tent
DMC
3.
05
4.3
155
2.
30
0.5
(a) 3s_vs_5z
vs Figure 26: Ablation studies on r- in SMAC tasks.
(b) 5m vs 6m
(c) 365 vs 3862
192 JCUS W71 DMU 1200, ml.
ESUJ KIRK W DiUl KCOA way?
0C6
80-
TenrVin 254
(005
002
1.90
(a) 3s_vs _V___ 5z
(b) 5m vs 6m
(c) 3s5z vs 3s6z
Figure 27: Ablation studies on rf in SMAC tasks.
D.11 COMPARISON OF EPISODIC INCENTIVE WITH EXPLORATORY INCENTIVE
In this subsection, we replace the episodic incentive with another exploratory incentive, introduced by (Henaff et al., 2022). In (Henaff et al. 2022). the authors extend the count- based episodic bonuses to continuous spaces by introducing episodic elliptical bonuses for exploration. In this concept, high reward is given when the state projected in the embedding space is different from the previous states within the same episode. detail, with a given feature encoder Φb the elliptical bonus by at timestep t is computed as follows:
b, 4 (. ((st)
(18)
where C is an inverse covariance matrix with an initial value of C = 1/Ne3a.. Here, Ae30 is covariance regularizer. For update inverse covariance, the authors suggested computationally efficient update as
1
c++ = CT
+ bt+1 un?
(19)
where u = C, ((5441). Then, the final reward F with episodic elliptical bonuses by expressed as
Fz rr + Beabb.
(20)
where Bess and rt are corresponding scale factor and external reward given by the environment, respectively.
For this comparison, utilize the dCAE structure as a state embedding function o. For mixer, QPLEX (Wang et al.) 20206) is adopted for all cases, and we denote the case with an elliptical incentive instead of the proposed episodic incentive as QPLEX (SE+E3B). Figure 28 illustrates
26

---

Published as a conference paper at ICLR 2024
EMU (SE+EI)
EMC
GPLEX
OPLEX (SE) E361
6mLys.cm
80
Took Win PS)
60
60-
0-
1.0
2.0
0.0
1.0
[(mi]]
((ml)
Figure 28: Performance comparison with elliptical incentive on selected SMAC maps.
the performance of adopting an elliptical incentive for exploration instead of the proposed episodic incentive. QPLEX (SE+E3B) uses the same hyperparameters with EMU (SE+EI) and we set
A035 0.1 according to Henaff et al. (2022).
As illustrated by Figure 28 adopting elliptical incentive presented by (Henaff et al. 2022) instead of an episodic incentive does not give any performance gain and even adversely influences the performance compared to QPLEX. It seems that adding excessive surprise-based incentives can be disturbance in MARL tasks since finding a new state itself does not guarantee better coordination among agents. In MARL, agents need find the proper combination of joint action in given similar observations when finding an optimal policy. On the other hand, in nigh-dimensional pixel- based single-agent tasks such Habitat (Ramakrishnan et al., (021), finding new state itself can be beneficial in policy optimization. From this, we can note that adopting a certain algorithm from single-agent RL case to MARL case may require a modification or adjustment with domain knowledge.
As a simple tuning, we conduct parametric study for 3033 {0.01, 0.1} adjust magnitude of incentive of E3B. Figure 29 illustrates the results. In Figure 29 QPLEX (SE+E3B) with Bess. = 0.01 shows a better performance than the case with Best 0.1 and comparable performance to EMC in 5m_ vs_ _mm However, EMU with the proposed episodic incentive shows the best performance. From this comparison, we can see that incentives proposed by previous work need to be adjusted
EMU (SE+E))
EMC
OPLEX Ras-001)
OPLEX (8E+E1B, Baao-011)
Sm 6in
00
00-
Tost Wh
00
00
80
T.0m0
Figure 29: Performance comparison with an elliptical incentive on selected SMAC maps.
according to the type of tasks, it was done in EMC (Zheng et al. 2021). On the other hand, with the proposed episodic incentive we do not need such hyperparamettr--calinng allowing much more flexible application across various tasks.
27

---

Published as a conference paper at ICLR 2024
D.12 ADDITIONAL TOY EXPERIMENT AND APPLICABILITY TESTS
In this section, we conduct additional experiments on the didactic example presented by (Zenng et al. 2021) how the proposed method would behave in simple but complex coordination task. Additionally, by defining Rear to define the desirability presented in Definition T we can extend
EMU single-agent RL task, where strict goal is not defined in general.
Didactic experiment on Gridworld We adopt the didactic example such as gridworld environment from (Zheng al. 2021) demonstrate the motivation and how the proposed method can overcome the existing limitations of the conventional episodic control. In this task, two agents in gridworld (see (0jure330aa00
Figure (0(a)) need reach their goal states the same time get a reward 10 and only one
arrives first, they get a penalty with the amount of -p. Please refer to (Zheng et al. 2021 for further
details.
Wall
G
Visible Zone
VON
0.0
0.1
02
U.
D4
05
(md)
(a) Gridworld
(b) Performance evaluation (p
Figure 30: Didactic experiments on gridworld.
To see the sole effect of the episodic control, we discard the curiosity incentive part of EMC, and for a fair comparison, we set the same exploration rate of c-greedy with T. 200 for all algorithms. We evaluate the win-rate with 180 samples (30 episodes for each training random seed and 6 different random seeds) at each training time. Notably, adopting episodic control with naive utilization suffers from local convergence (see QPLEX and EMC (QPLEX) in Figure b)), even though expedites learning efficiency at the early training phase. On the other hand, EMU shows more robust performance under different seed cases and achieves the best performance by an efficient and discreet utilization of episodic memories.
Applicability test to single agent RL task We first need to define Rev value to effectively apply EMU to single-agent task where goal of episode is generally not strictly defined, unlike cooperative multi-agent tasks with a shared common goal.
In single-agen task where the action space is continuous such MuJoCo (Toorov al. 2012). the actor-critic method is often adopted. Efficient memory utilization of EMU can be used to train the critic network and thus indirectly influence policy learning, unlike general cooperative MARL tasks where value-based RL is often considered.
We implement EMU on top TD3 and use the open-source code presented in (Fujimoto We begin to train the model after sufficient data is stored in the replay buffer and conduct 6 times of training per episode with 256 mini-batches. Note that this is different from the default settings of RL training, which conducts training at each timestep. Our modified setting aims to see the effect on the sample efficiency of the proposed model. The performance of the trained model is evaluated at every 50k timesteps.
2018).
We use the same hyperparameter settings as in MARL task presented in Table 8 except for the update interval, temb 100K according to large episodic timestep in single-RL. compared to MARL tasks. It worth mentioning that additional customized parameter settings for single-agent tasks may further improve the performance. In evaluation, three single-agent tasks such as Hopper-v4, Walker2D-va and Humanoid d-v4 are considered, and Figure 32 illustratees each task. Here, & = 1.3e 5 is used for Hopper - and Walker2D-v4, and dg = 1.3e 3 is used for Humanoid-as as Humanoid- 376-dimension. Please refer to Todorov al. 2012) for a detailed description of tasks.
task contains much higher state dimension space as
28

---

Published as a conference paper at ICLR 2024
(a) Hopper-v4
(b) Walkor2D-v4
(c) Numanoid-ve
Figure 31: Illustration of MuJoCo scenarios.
2000
TD3
2000
(TD3)
(TD33)
[NJITDD)
1500
1500
1000
1000
500
500
0
0
3.0
(a) Performance (Hopper)
(b) Performance (Malker2D)
(c) Performance (Humanoid)
Figure 32: Applicability test single agent task (Rurr 500).
In Figure 32 EMU (TD3) shows the performance improvement compared to the original TD3. Thanks to semantically similar memory recall and episodic incentive, states deemed desirable could have high values, and trained policy is encouraged to visit them more frequently. As result, EMU (TD3) shows the better performance. Interestingly, under state dimension as Humano d-vv task, TD3 and EMU (TD3) show distinct performance gap in the early training phase. This is because, in a task with a high-dimensional state space, it is hard for a critic network to capture important features determining the value of given state. Thus, it takes longer to estimate state value accurately, However, with the help of semantically similar memory recall and error compensation through episodic incentive, critic network EMU (TD3) can accurately estimate the value of the state much faster than the original TD3, leading to faster policy optimization.
Unlike cooperative MARL tasks, single-RL. tasks normally do not have desirability threshold. Thus, one may need to determine Rehr based on domain knowledge or a preference for the level of return be deemed successful. Figure 33
presents a performance variation according to Rehr-
103
EKU (TO2 5000
ENU (TD3 AT-
2000
BHU (T02AAu-1000 1000)
2000
EHU (TD3 10:0000
1500
1500
500
0
step
step
205
Hopper-v4
(b) Wa ker2d-v4
Figure 33: Parametric study on Rehr-
When we set Rer = 1000 in Walker2d task, desirability signal is rarely obtained compared to the case with Rehr 500 in the early training phase. Thus, EMU with Ruh 500 shows the better performance. However, both cases of EMU show better performance compared to the original TD3. In Hopper task, both cases of Rehr 500 and R 1000 show the similar performance. Thus,
29

---

Published as a conference paper at ICLR 2024
when determining Rehrr it can be beneficial set small value rather than large one that can be hardly obtained.
Although setting small Rehr does not require much domain knowledge, possible option to detour this is periodic update of desirability based on the average return value (s) in all s E Dg. this way, certain state with low return which originally deemed desirable be reevaluated as undesirable as training proceeds. The episodic incentive is not further given to those undesirable states.
Scalability to image-based single-agent RL task Although MARL tasks already contain high- dimension state space such 322-dimension in MMM2 and 282-dimension in corri dor, image- based single RL tasks, such as Atari Bellemare et al. (2013) game, often accompany higher state spaces such [210x160x3] for "RGB" and [2 160] for "grayscale", We use the "grayscale" type for the following experiments. For the details of the state space in MARL task, please see Appendix
In an image-based task, storing all state values to update all the key values in DE as fg updates can be memory-inetine memory-inefficient, and semantic embedding from original states may become overhead compared to the case without it. In such case, one may resort to pre-trained feature extraction model such as ResNet model provided by torch-vision in a certain amount for dimension reduction only, before passing through the proposed semantic embedding. The feature extraction model above is not an object training.
As example, we implement EMU on the top of DQN model and compare it with the original DQN on Atari task. For the EMU (DQN), we adopt some part of pre-trained ResNet18 presented by torch-vision for dimensionality reduction, before passing an input image semantic embedding. At each epoch, 320 random samples are used for training in Breakout task, and 640 random samples are used in Al ie task. The same mini-batch size of 32 is used for both cases. For fo training, the same parameters presented in Table 8 are adopted except for the temb = 10K considering the timestep of single RL task. We also use the same 82 1.3e 5 and set Rehr 50 for Breakout and Rehr = 40 for Al Alien, respectively. Please refer to Bellemare et al. (2013) and https: : / ggmmass um r ama. org/ envi ronments atari for task details, As in Figure we found a performance gain by adopting EMU on high- nigh-dimensional image-based tasks.
34
150
100
80-
20000 40
20
3200
10020 epars
15000
20020
5000
12000 eporh
15920
Breakout
(b) Performance (Breakout)
(c) Alien
(d) Performance ien)
Figure 34: Image-hased single-RI, task example.
30

---

Published as a conference paper at ICLR 2024
E TRAINING ALGORITHM
E.1 MEMORY CONSTRUCTION
During the centralized training, we can access the information on whether the episodic return reaches the highest return Rmax or threshold Rehr, i.c. defeating all enemies in SMAC or scoring a goal in GRE When storing information to DE, 3, by the definition presented Definition. Vs Tg-
set {(s) 1 for
For efficient memory construction, we propagate the desirability of the state similar state within the threshold 5. With this desirability propagation, similar states have an incentive for a visit. In addition, once memory saved in DE, the memory is preserved until becomes obsolete (the oldest memory to be recalled). When desirable state is found near the existing suboptimal memory within we replace the suboptimal memory with the desirable one, which gives the effect of a memory shift to the desirable state. Algorithm 2 presents the memory construction with the desirability propagation and memory shift.
Algorithm 2 Episodic memory construction
1: Ê Optimality of trajectory
2: T {so. ag To. $1,..., ST }: Episodic trajectory
3: Initialize R²- = 0
4: for
do Compute
5:
(st) and = (x: /a)/0s 0x
6: pick the nearest neighbor ā; E DE and get Dr. if 1lūe - < 8 then
7:
3rl12
8:
Noau f- (ot +1
9:
if ET == then
10:
(&t) + Ng (it) +
11:
end if
12:
& = and ÊT = then
& {τ
desirability propagation memory shift
14:
i, + It. Dt + yt. Se + St
15:
H, + Rs
else
17:
if H; < R4 then H, +- R.
end if
19:
end if
20:
else
21:
Add memory DE (01, Ve, Re, St,
22:
end
23:
end for
For memory capacity and latent dimension, we used the same values as Zheng et al. 202 and Table showw the summary of hyperparameter related episodic memory.
Table 6: Configuration of Episodic Memory.
Configuration
Value
episodic latent dimension, dim(z)
4
episodic memory capacity
1M
scale factor, À
0.1
(for conventional episodic control only)
The memory construction for EMU seems to require significantly large memory space, especially for saving global states s. However, Dr E uses CPU memory instead of GPU memory, and the memory required for the proposed embedder structure is minimal compared the memory usage of original
31

---

Published as a conference paper at ICLR 2024
Table 7: Additional CPU memory usage save global states.
CPU memory usage (IM data)
SMAC task
(GiB)
5m vs 6m
0,4
3s5z vs 3s6z
0.9
MMM2
1.2
RL training (<1%). Thus, memory burden due to a trainable embedding structure is negligible. Table presents examples of CPU memory usage save global states E DDg.
E.2 OVERALL TRAINING ALGORITIIM
In this section, we present details of the overall MARL training algorithm including training of fa. Additional hyperparameters related to Algorithm to update encoder fa and decoder foz are presented in Table g Note that variables N and B are consistent with Algorithm]
Table 8: EMU Hyperparameters for fa and lo training.
Configuration
Value
a scale factor of reconstruction loss, Arcon
0.1
update interval, tmmb training samples, N batch size of training, B
1K
102.4K
1024
Algorithm 3 presents the pseudo-code of overall training for EMU. In Algorithm 3
network parame-
ters related mixer and individual Q-network denoted as 8, and double Q-learning with target network is adopted as other baseline methods (Rashid et al. 2018 2020 Wang et al. 2020b Zheng
al. 2021
Chenghao al. 2021).
Algorithm 3 EMU: Efficient episodic Memory Utilization for MARL
1: D: Replay buffer
2: Dy: Episodic buffer
3: Q): Individual Q-network of n agents
4: M: Batch size of RL training
5: Initialize network parameters 0, d, s/
6: while fene S tmax do
7: Interact with the environment via e-greedy policy based on [Q) and get a trajectory T.
8: Run Algorithm 2 to update DE with
9: Append T to D
10: for
to Reirele do
11:
Get M sample trajectories Run MARL training algorithm using [T], and De. to update with Eg 10
M D
12:
13:
end for
14:
if
tene mod femb = 0 then
15:
Run Algorithm to update d, v
16:
Update all Dg with updated fø
17:
end
18:
end while
Here, any CTDE training algorithm can be adopted for MARL training algorithm in line 12 in Algorithm 3 As mentioned in Section C.4 training of fo and fo and updating all DD only
32

---

Published as a conference paper at ICLR 2024
takes less than two seconds most under the task with largest state dimension such corridors Thus, the computation burden for trainable embedder is negligible compared to the original MARL training.
F MEMORY UTILIZATION
A remaining issue in utilizing episodic memory is how to determine proper threshold value 8 in E Note that this 5 is used for both updating the memory and recalling the memory. One simple option is determining 8 based prior knowledge experience, such hyperparameter tuning. Instead, in this section, we present a more memory-efficient way for å selection. When computing || x||2 < 6, the similarity is compared elementwisely. However, this similarity measure puts different weight on each dimension of since each dimension of x could have different range of distribution. Thus, instead of z, we utilize the normalized value. Let define a normalized embedding with the statistical mean (#2) and variance (a₂)) of x
= Mz)/
(21)
Here, the normalization is conducted for each dimension x. Then, the similarity measure via has similar range of
ll - 1 < 8 with Eq. 21 puts equal weight to each dimension, distribution each dimension. addition, an affine projection of Eq. [2 maintains the closeness of original z-distribution, and thus measure the similarity.
safely utilize y-distribution instead of z-distribution to
In addition, y defined in Eq. 21 nearly follows the normal distribution, although it does not strictly follow it. This due to the fact that the memorized samples x in DE do not originate from the same distribution,
they uncorrelated, as they can stem from the same episode. However, achieve an approximate coverage of the majority of the distribution, specifically 3a, in both positive and negative directions of y. by setting 6
can
(2 30 Bo,y)dim(e)) (dim(y)
8 <
(22)
M
For example, when M 1e and dim(y) 4, if o, 1 then 8 < 0.0013. This is the reason select 0.0013 for the exploratory memory recall.
33


Reviews:
[{"review_id": "5XxzEopSkQ", "reviewer": "Reviewer_8Ngy", "summary": "This work presents a new framework for co-operative multi-agent RL that uses semantic memory embeddings to construct a novel reward structure that augments the environment reward by incentivizing desirable transitions. The framework, referred to as Efficient episodic Memory Utilization (EMU), comprises of an encoder-decoder network to learn semantically meaningful embeddings. The network is then utilized to obtain a reward that incentivizes desirable transitions. Experimental results in the benchmark Starcraft environments and Google Research Football demonstrate EMU’s superior performance to existing methods.", "questions": "1. Results for **1)** in Weaknesses. These set of results are not completely necessary but would be good to see. A well-reasoned argument about why the method should not be difficult to scale will also suffice. \n\n2. Results for **2)** in Weaknesses. \n\n3. It would be helpful to provide details about the state space, action space, environment reward and episode lengths for both SMAC and GRF in the Appendix. \n\n4. **Section 3.2** can use more intuition and better writing. It is unclear to me why the episodic inventive for a desirable transition is set to be proportional to the difference between the true value and the predicted value. Is this done to incentivize visits to states where the Q network has not converged?\n\n5. What are the implications of *Theorem 2*?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Clear writing and presentation:** Except for some minor subsections, the paper is generally well-written, easy to follow and presents a coherent story. \n- **Promising and extensive results**: The method outperforms existing works in standard benchmark domains. The work presents many experiments and ablation studies to analyze and demonstrate the effectiveness of different components of the proposes framework.", "weaknesses": "- **Scalability**: Based on the encoder-decoder architecture of the paper, I am assuming that the global state for the environments used is feature-based. It is unclear whether this method will scale to vision-based environments due to **a)** the memory requirements of storing many images, **b)** the optimization difficulty in reconstructing image-based states and **c)** the effectiveness of the introduced reward structure in high-dimensional state spaces. However, I acknowledge that many existing works in this area utilize feature-based observations. \nRegardless, it would greatly improve the strength of the results of this paper if some gains could be shown in vision-based environments.\n\n- **Evaluation**: It appears that random projection performs almost equivalently to EmbNet/dCAE when compared using test win rates. The introduction of a new metric (overall win-rate) highlights that EmbNet/dCAE enable faster/more sample-efficient learning. However, improvement on this new metric is not as significant as the original win rate (which is the standard benchmark in the community). I would be curious to see the curve for EMU with random projection added to **Sections 4.1** and **4.2** to better understand the significance of EmbNet/dCAE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XGUIrsORrq", "reviewer": "Reviewer_t4Jc", "summary": "This paper presents a new framework called Efficient episodic Memory Utilization (EMU) to effectively exploit episodic memory for cooperative multi-agent reinforcement learning (MARL). EMU mainly relies on two features: \n 1) A learned semantic embedding embedding that allows to easily pair similar states\n 2) An \"episodic incentive\" mechanism to select the most useful transitions from the buffer when learning", "questions": "Beyond my recommendations above regarding writing, there is a common abuse of \"the\" through the text, e.g. \"In spite of the required exploration in MARL with CTDE, ${the}$ recent works on episodic control emphasize the exploitation of episodic memory to expedite reinforcement learning. Episodic control (Lengyel & Dayan, 2007; Blundell et al., 2016; Lin et al., 2018; Pritzel et al., 2017) memorizes ${the}$ explored...\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper presents a novel framework and studies the effect of episodic memory in MARL, which to the best of my knowledge is an underexplored area. The theoretical foundation is good although sometimes it is difficult to grasp the motivation and intuition of some parts when first presented. The paper also includes a sound analysis of the performance of EMU in two relevant MARL settings with abundant ablations that help to understand the contributions of the different features.", "weaknesses": "The biggest weakness of this work at its current state is the limited scope of the literature review and the lack of comparison with existing methods for episodic memory. There are multiple works ([1-3] to name a few) that have created similar frameworks in single-agent settings, specially in exploration settings. So one wonders what prevents port these frameworks here? Without that for instance the embedding procedure of EMU could be a reinventing the wheel from existing procedures in [1,2]. I strongly encourage authors to visit that line of works and contrast those approaches with the features incorporated in EMU.\n\nMoreover, I believe that the comparison with related work is imperative to understand the position of the paper and its contributions and should not be relegated to the appendix. \n\n\nAs a minor issue, writing also should be reviewed, but clarity in general is good\n\n[1] Henaff, M., Raileanu, R., Jiang, M., & Rocktäschel, T. (2022). Exploration via elliptical episodic bonuses. Advances in Neural Information Processing Systems, 35, 37631-37646.\n[2] Le, H., Do, K., Nguyen, D., & Venkatesh, S. (2023). Intrinsic Motivation via Surprise Memory. arXiv preprint arXiv:2308.04836.\n[3] Fernandes, D. M., Kaushik, P., Shukla, H., & Surampudi, B. R. (2022). Momentum Boosted Episodic Memory for Improving Learning in Long-Tailed RL Environments.\n\n\n\n---- Post Second Rebuttal ---\n\nI want to thank the authors for the additional work to address the concerns, specially what you wrote in your last response from\n\"It seems that adding a surprise-based incentive can be\".... until \"For these reasons, incorporating the feature embedding structure from the single-RL domain and its learning framework into the multi-agent domain may necessitate extensive modification, resulting in a distinct line of research.\" was the kind of motivation and comparison that I was looking for. \n\nThis, together with the results in Appendix D.13 makes clear that the existing methods from single agent literature is not as good as EMU in this context and that indeed it is a relevant contribution that authors are giving to the community.\n\nMy only remaining comment is that none of this new work and the discussion is present in the main document, (at least I don-t see anything in magenta there) I would encourage the authors to incorporate a reference in the introduction highlighting that existing methods in single agent reinforcement learning are not valid here (incorporating a reference to this part of the appendix),  \n\nI believe now that the paper is sound and there is enough evidence to support the main claims from the authors. I am updating my score accordingly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pjw4Yc97L7", "reviewer": "Reviewer_7Wwz", "summary": "The paper introduces the Efficient episodic Memory Utilization (EMU) for cooperative multi-agent reinforcement learning (MARL). Addressing the challenges in MARL where agents often get trapped in local optima, EMU aims to accelerate learning by leveraging a semantically coherent episodic memory buffer and selectively promoting desirable transitions. EMU uses an encoder/decoder structure to train semantically coherent episodic memory and introduces an episodic incentive reward structure to enhance performance. The proposed method is evaluated on benchmarks like StarCraft II and Google Research Football, demonstrating its superiority over existing methods.", "questions": "1. Can the author explain how to set the return threshold for desirability in experiments? Is this threshold dynamic or fixed?\n2. Is the incentive reward only effective when a trajectory is desirable? Does this mean episodic memory is useful only when a very good trajectory is explored, which can be hard?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well motivated and well written. Particularly, its visual illustration of Figure 2 and 3 are helpful.\n2. Although the proposed idea of semantically coherent memory looks simple to use an AE structure, it seems not being explored in multi-agent settings. One potential related work is generalized episodic memory, which can also be regarded semantically coherent episodic memory. \n3. The episodic incentive is interesting and looks effective.\n4. The proposed method shows strong empirical results. Its ablation studies are extensively conducted.\n5. This paper conducts a sufficient review of related work and is well positioned.", "weaknesses": "1. EMU needs to set a return threshold to determine the desirability of a trajectory. This may require some domain knowledge to properly determine it, even when using R_{max}. This knowledge may partially explain its outperformance. \n2. When the key encoder is updated, the proposed method needs to update all keys in the memory, which seems quite computationally intensive. \n3. It may be interesting to compare the proposed method with MAPPO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6RXiz6evZq", "reviewer": "Reviewer_mSGj", "summary": "This paper aims to improve the efficiency in multi-agent reinforcement learning (MARL). It leverages episodic memory and introduces episodic incentive to help exploring desirable trajectory. This paper demonstrate both theoretical analyses and empirical results.", "questions": "* See weakness\n* The approach adopts a state embedding instead of random projection. Does this make the approach more hard to converge?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* This paper provides comprehensive theoretical analyses and strong performance improvement. The paper proves that the approach can help policies converge to the optimal policies. The paper also shows the great performance in Google football and StarCraft.\n* This paper is well-structured and written. The paper provides full details about the method and the experiment. It also constructs detailed ablation studies.", "weaknesses": "* I have concerns about the desirable trajectory. In paper, the author set $R_{thr}=R_{max}$. Since the desirable trajectories are the states that can achieve maximum returns, they must be the optimal states. What if the agents are impossible to achieve $R_{max}$. How to determine $R_{thr}$ in other environments?\n* The dimension of $x$ is very small (i.e. 4 according to table 4 in the appendix). It's doubtful that it can reconstruct the global state.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)