IA-MARL: Imputation Assisted Multi-Agent Reinforcement Learning for Missing Training Data
Authors: Dongsun Kim
Sinwoong Yun
Jemin Lee
Eunbyung Park
Venue: ICLR 2022 Submitted (2022)

Abstract: Recently, multi-agent reinforcement learning (MARL) adopts the centralized training with decentralized execution (CTDE) framework that trains agents using the data from all agents at a centralized server while each agent takes an action from its observation. In the real world, however, the training data from some agents can be unavailable at the centralized server due to practical reasons including communication failures and security attacks (e.g., data modification), which can slow down training and harm performance. Therefore, we consider the missing training data problem in MARL, and then propose the imputation assisted multiagent reinforcement learning (IA-MARL). IA-MARL consists of two steps: 1) the imputation of missing training data, which uses generative adversarial imputation networks (GAIN), and 2) the mask-based update of the networks, which trains each agent using the training data of corresponding agent, not missed over consecutive times. In the experimental results, we explore the effects of the data missing probability, the number of agents, and the number of pre-training episodes for GAIN on the performance of IA-MARL. We show IA-MARL outperforms a decentralized approach and even can achieve the performance of MARL without missing training data when sufficient imputation accuracy is supported. Our ablation study also shows that both the mask-based update and the imputation accuracy play important roles in achieving the high performance in IA-MARL. 

Under review as a conference paper at ICLR 2022
MARM:::: IMPUTATION ASSISTED MULTI-AGENT REINFOR LEARNING FOR MISSING TRAINING DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, multi-agent reinforcement learning (MARL) adopts the centralized train- ing with decentralized execution (CTDE) framework that trains agents using the data from all agents at centralized server while each agent takes an action from its observation. In the real world, however, the training data from some agents can be unavailable at the centralized server due to practical reasons including communication failures and security attacks (e.g., data modification), which can slow down training and harm performance. Therefore, we consider the missing training data problem in MARL, and then propose the imputation assisted multi- agent reinforcement learning (IA-MARL). IA-MARL consists of two steps: 1) the imputation of missing training data, which uses generative adversarial imputation networks (GAIN), and 2) the mask- based update of the networks, which trains each agent using the training data of corresponding agent, not missed over consecutive times. In the experimental results, we explore the effects of the data missing proba- bility, the number of agents, and the number of pre-trale trainiiiinng episodes for GAIN on the performance of IA-MARL. We show IA-MARL outperforms a decentralized approach and even can achieve the performance of MARL without missing training data when sufficient imputation accuracy is supported. Our ablation study also shows that both the mask-based update and the imputation accuracy play important roles in achieving the high performance in IA-MARL.
INTRODUCTION
Reinforcement learning (RL) solves many challenging problems including the game playing (Mnih) al. 2015) and the robot control (Levine et al. 2016), which focus on the single-agent RL envi- ronment, modeled as the Markov decision process (Sutton and Barto 2011). However, there exist many
problems that involve interaction among multiple agents such as multi-robot control (Hürtenrauch et al. 2019) and multiplayer games (Silver et al. 2017] Bard et al. 2020). Hence, the multi-agent reinforcement learning (MARL) that operates multi-agent domain has been introduced and now becomes one of the most active and challenging RL research areas.
In MARL, the decentralized approach has been used to train each agent based on its trajectory (Tan) 1993). However, it often shows unstable and low performance due to non-stationary environment and partially observable information (Tan 1993 Foerster et al., 2017) that inherits from the decen- tralization. Specifically, as the agents evolve their policies independently, the environment becomes non-stationary, which unstabilizes training at each agent. In addition, the agent may not observe the information of other agents, which causes low performance in the cooperative or competitive environment (Lowe et al. 2017).
Recently, the centralized training with decentralized execution (CTDE) framework has been intro- duced for MARL (Olichock et al., 2008 Foerster et al. 20188.. This can alleviate the non-stationary environment and the partially observable information problems (Lowe et al.) 2017) and encourages coordination among agents (Foerster et al. 2018). In the execution CTDE, each agent takes an action based on its observation, while the training of the agents is performed at a centralized server after collecting observations, actions, and rewards of all agents. In existing works, those data from all agents assumed to be available at the centralized server, which may not be always true in reality.

---

Under review as a conference paper at ICLR 2022
The data from distributed agents can be unavailable due to practical reasons including the communi- cation failure, hardware limit, and security attacks (Lakshminaryyan et al.. 1999; Twala| (2009). For instance, in wireless sensor applications of MARL such vehicle tracking (Liang et al. 2020) and environmental monitoring (Li et al.) 2020), sensors (i.e., agents) transmit their sensed information to receiver (i.e., centralized server) for training. The training data, transmitted from sensors, can be missed when the communication is unstable. In addition, even when the training data successfully arrives at the centralized server, certain data can be removed from the training dataset due security attacks such as false data injection (Yan et al., (016)) and unauthorized data modification (Ferretti al. 2014). As one can readily imagine, this missing training data can cause a serious problem in MARL as the training cannot be performed.
One possible solution on this missing data problem is to use only the training data that contains data from all agents without missing. However, in this case, the number of training data can dramatically decrease as the number of agents increases or the data missing happens more often, which delays the training. Another solution can be the use of imputation for replacing the missing training data in the MARL. However, the training data of this case can be different from the original data, which potentially degrades the performance. Therefore, as discussed above, the missing data problem should be carefully considered for bringing the MARL to the next level for wider range of applications. Despite of it, the best of our knowledge, the missing data problem in MARL has not been taken into account in existing works.
In this paper, we propose imputation assisted multi-agent reinforcement learning (IA-MARL) with considering the missing training data problem, where the training data of each agent, consists of observation, action, and reward, can be randomly missed with certain probability. The proposed IA-MARL consists of two steps: 1) the imputation of missing training data and 2) the mask-based update of the networks. Specifically, for the imputation of the missing training data, we use a generative adversarial imputation network (GAIN) to impute the data from all agents, and we form the data for the training of agents. We then perform the mask-based update, which trains the value function and the policy of each agent by selectively using the training data of the corresponding agent, not missed over the consecutive times. In the experimental results, we show the IA-MARL outperforms decentralized approach and also can achieve the performance of MARL with all training data without missing. We then also show the performance of IA-MARL for different missing probabilities, the number of agents, and the number of pre-training episodes for GAIN. From the ablation study, we also verify the importance of the mask-based update as well as the imputation accuracy in multi-agent environments when the training data can be missed.
2 RELATED WORK
Independent Q-learning has been proposed as a decentralized approach to train each agent using its own data independently (Tan) 1993). The independent Q-learnin is used for the tabular environment [Littman] 1994), and deep learning-based approaches are presented in Tampuu et al.) (2017); Gupta al. (2017). 7). Independent Q-learning, however, suffers from the non-stationary environment and partially observable information problems.
CTDE is one of the solutions for those problems. In CTDE, the data from all agents are used for the centralized training, while the execution at each agent only requires its observation (Olichock et al.) 2008). For instance, the centralized server trains the value function of each agent using observations, actions, and rewards of all agents, while each agent takes an action based on its observation (Lowe
al. 2017).
Recently, MARL algorithms that adopt CTDE framework have been presented. For instance, Lowe et al. (017)) proposes the multi-agent deep deterministic policy gradient (MADDPG) that extends deep deterministic policy gradient (DDPG) for the continuous control of multiple agents. For the credit assignment problem, the counterfactual baseline (Foerster et al. 2018) and the value function factorization that determines the contribution of each agent are used (Sunehag et al.) 2018 Rashid et al. 2018; Son et al.) 2019). To improve the performance of MARL, the soft value function and the multi-head attention used in Igbal and Sha (2019), and the communication between agents that provides additional information to each agent is introduced in Foeesttr et al.] (2016); Mordatch and Abbeel (2018) while the limited communication channel between agents during the execution
2

---

Under review as a conference paper at ICLR 2022
is considered in Kim et al. (2019) to address the real world communication channel constraints in MARL. However, none of the prior work considers the missing training data problem.
Imputation is the research area that replaces missing data, which has been used for many applications including medical data and image concealment (Rubin 2004). For the imputation, some techniques such multivariate imputation by chained equations (MICE) (Buuren and
2010), matrix completion (Mazumder et al.) 2010). and MissForest (Stekhoven and Bühlmann 2012) are proposed. However, when the imputation is used for the data that have large data space (e.g.. the data obtained from agents in CTDE), the techniques with insufficient expressive power might result low performance. For this case, some imputation techniques that have more expressiveness by adopting the deep neural networks can be more suitable such as the multiple imputation using denoising autoencoders (MIDA) (Gondara and Wang 2018), the bidirectional recurrent imputation for time series (BRITS) (Cao et al. 2018), and the generative adversarial imputation network (GAIN) (Yoon et 2018).
BACKGROUND
We consider decentralized partially observable Markov decision process, defined by tuple (S, A, P. 1r,2, S2, O, 7, n). where S, A, and Ω are set of states, actions, and observations, respectively. r and y are the reward and the discount factor, respectively, and n is the number of agents. We use 8 E S, E A, and E Ω for a state, an action, and an observation, respectively. We use subscript i for the corresponding agent and / for the time, e.g., Oi,r is the observation of agent at time We use bold symbols to denote observations, actions, and rewards of all agents, e.g., a = (01,2, ... (n,t)- Here, ((st+1/$t, ar) and ((,,(st) are the transition probability and the conditional observation probability, respectively.
3.1 DDPG AND MADDPG
The objective of the agent in the environment is to maximize the cumulative reward R, Ex= 7 Tr.. For this, we use the actor-critic method. The expected cumulative reward for given action and state is (st, a.) B[R,|s 8t, a az), which is called function or value function. Using the Bellman equation, the value function can be rewritten as Q(st, a,) s 2041 ar++1 Ir, ++ 1141)]. When the parameter @ is used for the value function approximation, the value function Q can be learned by minimizing the loss L()), given as
action-value
((0) = E [(Qo(s:. le) u) y + (Qo(se41,
(1)
DDPG is the widely-used choices for the policy update. The policy parameterized by Φ takes state s input and outputs deterministic action (s) in DDPG, where the gradient of is given as ( st) V Qo ())),=)s(a)))
(Φ) = V
(2)
MADDPG an algorithm that uses DDPG in the CTDE framework (Lowe et al. 2017). The value function in MADDPG takes observations and actions of all agents as input. Meanwhile, the policy of each agent takes its observation as an input since the agent can access to its observation only. In CTDE, the loss for the value function and the gradient for the policy are given as
((0:) R[(Qo, (or, ar) - -) y rie 7Q, (or+1, a(+1), e. Ps, Here, each agent has different value function and a policy, parameterized respectively as 0, and Φ--
(3)
V v, [(x)
Qo, (o, a at)lo..=m ro, (o, (ou.))
(4)
3.2 IMPUTATION
In statistics, imputation is used to replace missing data with substituted values. In order denote the missingness of the data, the mask M that has the same dimension as the data is used: the k-th component of the mask m4 O, when the k-th data va is missed, and m& = when xg is not missed. We denote the obtained data as M O X, where @ is elementwise multiplication. In the
imputation, the completed data X is given as
X + (1 M) o x
(5)
3

---

Under review as a conference paper at ICLR 2022
where X is the imputed data using the imputation algorithm including GAIN. For the loss of the neural network that imputes data, the mean squared error is used, given as
(x, Σ mx xx)².
(6)
Here, Ey is k-th component of the imputed data. Note the loss can be estimated when my = 1 since Jy can used only when it not missed.
METHODS
In this section, after introducing the missing training data problem in MARL and describing the imputation method, we propose IA-MARL.
4.1 MISSING TRAINING DATA PROBLEM AND IMPUTATION METHOD
We consider the MARL environment where the training data of cach agent is randomly missed, called the missing training data problem. Specifically, the training data from agent at time t, Ti = (o dj T,,)), is missed with the probability Pm,., = P(m = 0) < 1. as well. where mi,1 E {0, denotes whether the data of agent at time t is missed (m,, 0) or not (m, 1).
In the presence of missing data, to obtain completed data of all agents at time t for training, we first set the data for the imputation and the mask, respectively,
X = (T--1, Te, Te+1), M, = (m,-1, my Mc+1),
(7)
where T; = (Ti.ss T. Tn,t) and m, (I.,..J..mm..-
mn, JI. Here, | a cardinality of set and J|,[7] is an all-ones vector with length [T]. Note that for the accurate imputation of fi.,, the temporal data correlation as well as the data correlation across different agents should be used
For the imputation, we use GAIN (Yonn et al. (018)) as it has sufficient expressiveness to impute multi-agent data, does not require original data for the training, and also has state-of-art performance. The generator G' in GAIN takes the obtained data, the mask matrix, and the random matrix as inputs and outputs the imputed data. The imputed data and the completed data can be presented, respectively,
== G(M @ X₁, Me (1 (1) @ Ze),
(8)
X, - o X, + M,) o X.
(9)
where Z, is a random matrix that has the same dimension X,- In equation 9 the completed data can be represented as
Algorithm 1 GAIN Training and Imputation
Require:
Dataset DGAIN contains X, and M, 1: Generate random matrix Zy and hint matrix
X, = (f-11 9., T++1). The discriminator
D in GAIN takes X, and the hint matrix
2: Generate completed data Xt using equation
Ht as inputs and outputs the probability of being masked for the completed data as
3: Discriminate completed data using equation 10
4: Estimate Ln using equation 11 and update dis- criminator
Mr D(,, He),
(10)
5: Generate random matrix Z, and hint matrix He
H. = B, の Mr + 0.5(1 - B,).
6: Generate completed data X, using equation 9
7: Discriminate completed data using equation 10
8: Estimate LG using equation 12 and update genera- tor
In equation 10 B, (b1.--1..... bn) is a random sequence of 0 and 1, where Ph P(bb,. 1) is the hint probability.
9: Return X₁
We do not consider the case of missing all training data of all agents and time since the imputation cannot proceed. We also note that the proposed IA-MARL can solve the problems of missing certain part of 7.,, (e.g.. missing o;, 0q,()
), t' > 1. As increases, X. contains more information which may increase the imputation accuracy. At the same time, as t' increases, the input and output dimensions of GAIN increase which may decrease the imputation accuracy. Due to the ambiguous effect of t on the imputation accuracy, we left design choice and use t
"One may use Xx = (Tr.-.r... ),
4

---

Under review as a conference paper at ICLR 2022
The role of the hint matrix is to make the discriminator focus the component with b,.7 0. The objective of the discriminator D is to maximize the probability of estimating the mask matrix correctly, while the objective of the generator G' is to minimize the both accuracy of the discriminator and the loss of the imputation. Therefore, the losses of D and G are, respectively, given by
LD M.),,
(m log(m + (1 ma,,)) log(1 mast))
(11)
+
Lo E(X,,M))DDNN
Σ +
(1 mi,)) log((nn)) aGLM
(12)
where DEAIN is the dataset that contains obtained data and its mask matrix, and m,, [0, is
the probability of being masked, i.e., the i-th component of M, in equation 10 In equation 12 (X,, X2) is the difference between the imputed data and the obtained data as in equation 6 The parameter ag controls the importance of the imputation loss, which a hyperparameter. The training and the imputation of GAIN are given in Algorithm
4.2 IA-MARL: IMPUTATION ASSISTED MULTI-AGEN REINFORCEMENT LEARNING
In this subsection, we propose IA-MARL, which uses the completed data obtained from GAIN and the mask-based update that stabilizes the training in MARL for the missing training data problem. For
1-1
(=2
4=3
t=
(=5
6
DEAIN
Agent
X
71,2
X
+1.
T1,6
IA-MARL, we use MADDPG, but other actor- cetor-critic RL methods including policy gradient are also ap- plicable for MA-R. Note that with some modifi- cation on the value function update. IA-MARL can
Ageut
2
F2.1
Tap2
X
T2.1
x
Imputation
also be used for the value-based MARL algorithms including VDN and QMIX.
F
42
F3
ft
P,1
f1.2
f1.3
f1,4
f1.5
f1,6
In IA-MARL, the value function and the policy of agent i can be updated using the following loss and gradient.
Agent
F2.1
f2.2
+2.3
f2,4
fo.s
f2.6
[(, E [(Qo (or, -3
D C(0)), V H(h)
y = fú, + QQ (0.+1, ac+1),
(13)
D2 L((2). ((bz)
(o.) = E E,ND [ Map (ar))
(14)
Figure 1: An example of the mask-based update in IA-MARL when n = 2.
Var, Qo, (ò,, āz) mag(6600)
where 6/ and of indicate the target network parameters periodically updated as in (Hasselt et al. 2016). Even though we use imputation, the completed data may have different value with the original data, which harms the performance. To mitigate this, we propose the mask-based update that trains the value function and the policy using D. which is the set of the completed data for the training of agent i, given by
D. M 1,,E E,, , 13} U (
Mi 1},
(15)
where Tmax is an episode length. Note that D, in equation 15 contains data collected across episodes, and fr... is included in D. when Mi.T.... 1 to update 0; and Φb at time Tmax The mask-based update means, in the training of agent the completed data at time & used only when the data of agent is not missed over two consecutive times, and + 1. An example of D, is presented in Fig. As shown in Fig. only the completed data at time 2 and 5 are included in Dj. and the completed data at time 1 is included in D...
In MARL. the imputation and the mask-based update are used to make the policy and the value function (i.e., As. (ō,,)) and Qo, (ô,, â,)) similar to the ones without data missing (i.e., No, (ai. (o..)) and Qo, (Or, at)), respectively. In case of the policy, as mi,t = 1 (i.e., the data of agent
5

---

Under review as a conference paper at ICLR 2022
time exists) by the mask-bused update, value function, since mi, = by the mask-based update, the value function can be updated when (oi,s di,t. fi,t: O 0i,t+1 1) exists that mainly affect the update of Qo, (ô,, â,). Hence, if the imputation on the missing data of other agents accurate, Qo, (Ór, âz) can be similar to Qo, (o,, ac). We also show the importance of the mask-hased update and accurate imputation in Section 5.3.
have μo, (ds,,lōs.e) Mao (as,elos,)). In case of the
The training algorithm for IA-MARL is provided in Algorithm 2 In IA-MARL, GAIN is periodically trained and outputs completed data using Algorithm where
Algorithm 2 IA-MARL
1: Initialize DAAIN and GAIN G, D
2: Initialize di, 6/ 02 0. D; Vi
the dataset for GAIN, DAIIN is initialized
3: for episode to
to prevent the overfitting of GAIN the tra- jectory of outdated policies. We pre-train
4: if episode = Nue then
Initialize DGAIN Dj, Oi d. 0, end if
GAIN through pre episodes for better im- putation accuracy. After Ppc episodes,
6:
7: Initialize environment and receive o
we initialize the parameters all agents,
8:
for / = 1 to max do
ie., Φs. o/ 0g, 8/,, and D,, Vi, to prevent the agent from training with inaccurate data. The components in Algorithm can be modified according the MARL al- gorithm, where we use MADDPG in this work. Therefore, the training procedure follows that in Lowe et al. (2017 e.g., the action is selected as a, Us. (o:) N, where N is noise for exploration.
9:
For each agent i, select action as Store o,
10:
in DGAIN
11:
for all agent do
12:
Train Φ; and 0g using equation 13 and
equation 14
13:
Update target network of and 0g end for
14:
15:
end for
16:
Train GAIN using DOAIN and update Dr, then
initialize DGAIN
In IA-MARL, the number of training data is smaller than the number of obtained data
17: end for
due to the mask-based update. When the missing probability of agent over time is equal, i.c., Pm... Pm, Vt, and the number of obtained data is Np. the number of completed data for the training of agent given as
E [.I]
ND Pm.)
(16)
since the condition in equation T5 is satisfied when the data of agent exists during the consecutive times. Note that E [[D:]] decreases with Pm,
When the imputation is not applied, the centralized server can only use the data which is not missed from all agents over consecutive times, i... Dp = {T. m = Vi, E {1,... .Tax 1}} U {Tr.. MM.Tmm. Vi]. In this case, the im- putation is not required for the training. Hence, for given ND, the number of the training data is given
E E[Dr>]] = No II Pm,)
(17)
Therefore, the number of training data without imputation, [[[Dpp], decreases exponentially with number of agents n while the number of training data for IA-MARL, [[[D. not affected by We show the performance of MADDPG without imputation in Appendix B.
5 EXPERIMENTAL RESULTS
In this section, we provide MARL environment and hyperparameters, and then show the performance of TA-MARL with different missing probabilities, the number of agents, and the number of pre- training episodes for GAIN. Furthermore, we provide the ablation study which shows the importance of the mask-based update and accurate imputation in IA-MARL
"The code is in the supplementary materials and will he published after the review.

---

Under review as a conference paper at ICLR 2022
-20
-20
avergg rewvi73.
average rewarrs
-20
B1 10"
-40
Nuv 1 100
Nue 1 1x1
Mave x 10
(1r 4.x 102
Nuv x00
Nuw 4 1G
Myve 4 10?
-22
-40
Nas-
Nuu 0
Mac- 0
-150
MADDPS, Pw
-100
MADDKS, Ar.
-302
MADOPG, Phr
MADUPS, Pry DDPG
DDPG
DDPG
DDPG
-328
15000
-10
3868e 1909
-223
19339 cpisodes
20003
10000 2000 cotaades
opisodes
opisoces
(a)
(b)
(c)
(d)
Figure 2: Speaker-Listener Listener environment with a pair of speaker- listener when different training data missing probability pm and number of pre-training episodes Nur considered. The training data
missing probability is (a) Pm = 0.1, (b) Pm 0.2, (c) Pmm = 0.3, and (d) pm = 0.4. Figure best viewed in color.
5.1 ENVIRONMENT AND HYPERPARAMETERS
Among widely-used MARL benchmark environments including the multi-agent particle environments
(MPE) (Lowe et al., 2017 Mordatch and Abbeel, 2018) and the starcraft multi-agent challenge (SMAC) (Samvelyan et al. 2019), we use and modity the MPE that contains mixed cooperative and competitive environments. Note that SMAC is not used in experiments since MADDPG, adopted in IA IA-MARL, generally does not perform well in SMAC (Papoudakis et al. 2021).
The speaker-listener environments are composed of several pairs of agents, where each pair has speaker and listener, and landmarks have unique colors. In the environment, the speaker observes the target color and sends message to the listener. The listener then moves after observing the locations of landmarks and the message. Each speaker-listnner pair gets shared rewards, which is determined by the distance between the listener and the target landmark. Note that the speaker should learn which message to send, and the listener, who does not know the target landmark, should learn the message. We set the number of landmarks as 2 + n2/2 so that the number landmarks increases with the number of pairs (i.e., 2//2).
The tag environments are composed of one prey and multiple predators, where the number of predators is set 3, 4, and 5 (which means the number of agents are 4, and 6, respectively). In the environment, the objective of the prey is to run away from the predators, and the objective of the predator to catch the prey. Each time when the prey collides with the predator, predators get shared reward while the prey is penalized.
We evaluate the performance of the algorithm across 10 runs with different random seeds and the shade in graphs represents the confident interval. We use MADDPG for IA-MARL. As baselines, we use MADDPG and DDPG without missing training data (i.e., Pm = 0). We use the same hyperparameters in Lowe et al. (2017), except for the training frequency of the network parameters IA-MARL. Specifically, while MADDPG updates every 100 samples and collects 1024 samples before making an initial update, IA-MARL updates every 100/(1 Pmn)² samples and collects 1024/(1 )2 samples before making an initial update. Accordingly, the target network parameters are less frequently updated in IA-MARI. In the ppeaker-listenes environment, we evaluate the performance of IA-MARL against MADDPG speaker and listener. In the tag environment, we evaluate IA-MARL against MADDPG predators and DDPG prey.
5.2 PERFORMANCE OF IA-MARL.
Figure shows the average rewards as a function of the episodes with different number of pre- training episode and training data missing probability in the speaker-listener environment with one Pm 0.1, 0.2, 0.3, 0.4 is used, respectively. We observe that the performance IA-MARL outperforms the decentralized approach, i.e., DDPG, and also can achieve the performance of MARL without missing training data, i.e., MADDPG with Pm O. when GAIN is pre-trained sufficiently. Specifically, IA-MARL achieves the performance of MADDPG with any Npr for Pm 0.1 and Npue 2 4 x 10a for Pm 0.2 and 0.3. When pm 0.4, IA-MARL could not achieve the performance of MADDPG, but it is expected to achieve the similar performance of MADDPG with larger N ppre- Note that as Pm increases, mainly due the smaller number of
speaker-listena pair. In Figs. 2a 2d
7

---

Under review as a conference paper at ICLR 2022
100-00 1wc per
Mow - 330 - 0.1 Mos 11.-1.1.31, i
New 1x 12f,
witner, Pw DoPG
df Srm
-00
-10
-120
10300
20065 3006
19000
1000
2000 2000
1.00
1800
2000
200W
6008
1300
2000
2000
(b)
(c)
(d)
Figure 3: (a) Speaker- Listener environment with two pairs of agents. (b)-(d) Tag environment, where the upper graphs show the average rewards of the prey, and the lower graphs show the average rewards of the predator. The number of predators is in (b), in (c), and 5 in (d).
training data as shown in equation [6 IA-MARL slowly achieves the performance of MADDPG. Furthermore, as Pm increases, more pre-training episodes are required since GAIN is less trained due the larger value of imputation loss. When Nre small, large imputation error in completed data causes IA-MARL to hardly achieve the performance of MADDPG.
Figure shows the average rewards as function of the episodes with different pre-training episode and missing probability. In Fig. 3al the average rewards are shown for two pairs of speaker- we show the average rewards when the number of predators 3, 4, and 5, respectively, where the upper graphs show the rewards of the prey and the lower graphs show those of predators. observe IA-MARL can achieve the performance of MADDPG, and more pre-training and training episodes required Pm increases. In Fig. 3al when Pm 0.1 and 0.2, IA-MARL achieves the performance MADDPG slowly. Furthermore, Figs. [3b also show IA-MARL achieves the performance of MADDPG when pm 0.1, where the rewards of predators steadily increase, and the rewards of the prey decrease. However, as Pm increases, IA-MARL. cannot achieve the performance of MADDPG due to large imputation error. For instance, in Fig. 3al when Pm = 0.3. the performance of IA-MARL is in between the performance of MADDPG and DDPG. Similarly, in Figs, 3c and 3d when Pm. 0.2 and 0.3, IA-MARL cannot achieve the performance of MADDPG.
In Figs. 36 3d.
We also observe as the number of agents in- creases, more Npw is required make IA- MARL achieve the performance of MADDPG.
In the speaker-listener environment with Pm 0.3, Npw = 4 x 103 is required for n = 2 (see Figure 2ap. while Nuw > 10 is required for (see Figure 3a). Similarly, in the tag envi-
average rewards
-100
MADDPG, =
-150
DOPG
IA.MARL[GAIN,, 0.1
200
[A MARLLrandddm' Py 0.1
ronment with Pm 0.2, Npr = 10a is required for n = (see Figure 3bb. while Np > 1 is
Simpe-IA MARL(GAIN), Pa 0.1
Simp e MARLrandomm)
required for n and n (see Figures 3c 3d) The main reasons to require more N for the larger number of agents are 1) environ- ment with more agents becomes more complex and 2) the generator and the discriminator in GAIN require more training data as the input and output spaces increase with the number of agents.
2000
4000
6000
8000
10000
episodes
Figure 4: Speaker-Listence environment with a pair of agents. The average rewards with/without the mask-based update and with GAIN/random imputation are shown. We use Npr 10a for GAIN.
8

---

Under review as a conference paper at ICLR 2022
5.3 ABLATION STUDY AND LIMITATION
We compare the performance of IA-MARL with and without the mask-based update and with and without GAIN imputation. For simplicity, we call IA-MARL without the mask-based update as simple-IA-MARL. In simple-IA-MARLL, agent i is trained using the completed data D = { | E {1..... Tmax 1}}, which is different from IA-MARL that trains agent i using data D., When random imputation replacing missing training data with uniform random
GAIN is not used, variables, i.e., Os.ee ~ (mmmnn,,,es (Oi,e), m (Os,))), where 92 is the set of all Oi,t- Note the random imputation can be regarded as simple imputation method that has low imputation accuracy. We show the performance of IA-MARL and Comple-IA-MARE with different imputation method and missing probability in Appendix B.
Figure 4 shows the ablation study of IA-MARL when pm 0.1 and Nrr = 104, Firstly, we observe the average rewards of simple-IA-MARL. are even lower than that of DDPG for all imputation methods. The low performance of simple-LA L-- -MARL is due to the effect of imputation eπror on the gradient estimation, which is alleviated in IA- IA-MARL by the mask-based update. We also observe that the average reward of IA-MARI. with random imputation is similar or lower than that of DDPG. Since the accuracy of random imputation is low, both the MARL with/without the mask-based update cannot achieve the performance of MARL without missing data. Similarly, when the number of pre-training episodes for GAIN, Nrre. is small, the imputation accuracy is low, so the performance of IA-MARL is degraded as shown in Figs. 23 Therefore, when the imputation is not sufficiently accurate, the performance of IA-MARL can be low.
6 CONCLUSION
We propose IA-MARI. for the training of agents in the presence of missing training data. The key idea to use the imputation for replacing the missing data and the mask-based update that selectively uses the training data for each agent. In IA-MARL, MARL algorithm. In the experimental results, we verify the performance of IA-MARL for different training data missing probabilities, the number of agents, and the number of pre- pre-training episodes for GAIN. We show that A--AL with missing training data achieves comparable performance with MADDPG without missing training data, when GAIN is pre-trained sufficiently. Through the ablation study, we also show the importance of the mask-based update and the imputation accuracy in TA-MARL for achieving high performance.
use GAIN for the imputation and MADDPG for
REFERENCES
Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280: 103216, 2020.
S van Buuren and Karin Groothuis-Oudshoon mice: Multivariate imputation by chained equations in R. Journal of statistical software, pages 1-68, 2010.
Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. BIRTS: Bidirectional recurrent imputation for time series. In Advances in Neural Information Processing Systems, pages 6775- 6785, 2018
Luca Ferretti, Fabio Pierazzi, Michele Colajanni, Mirco Marchetti, and Marcello Missiroli. Efficient detection unauthorized data modification in cloud databases. In IEEE Symposium on Computers and Communications (ISCC), pages 1-6, 2014.
Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pages 2137-2145, 2016.
Jakob Foerster, Nantas Nardelli. Gregory Farquhar, Triantafyllos Afouras, Philip HS Torr, Pushmeet Kohli, and Shimon Whiteson. Stabilising experience replay for deep multi-agent reinforcement learning. In International Conference Machine Learning, pages 1146-1155. 2017.

---

Under review as a conference paper at ICLR 2022
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence, pages 2974-2982, 2018.
Lovedeep Gondara and Ke Wang. MIDA: Multiple imputation using denoising autoencoders. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 260-272, 2018.
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pages 66-83, 2017.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q- learning. In AAAI Conference on Artificial Intelligence, pages 2094-2100 2016.
Maximilian Hüttenrauch, Sosic Adrian, Gerhard Neumann, et Deep reinforcement learning for swarm systems. Journal of Machine Learning Research, 20654):1-31, 2019.
Shariq Iqbal and Fei Sha. Actor-attention-criiie for multi-agent reinforcement learning. In Interna- tional Conference on Machine Learning, pages 2961-2970, 2019.
Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning to schedule communication in multi-agent reinforcement learning. In International Conference on Representation Learning. 2019.
Kamakshi Lakshminarayan, Steven A Harp, and Tariq Samad. Imputation of missing data in industrial databases. Applied intelligence, 11(3):55--275 1999.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(1):13341173,, 2016.
Xinge Li. Xiaoya Hu, Rongqing Zhang, and Liuqing Yang. Routing protocol design for underwater optical wireless sensor networks: A multiagent reinforcement learning approach. IEEE Internet of Things Journal, 7(10):9805-99) 98188880009888888111818181111888 2020.
Teng Liang, Yan Lin, Long Shi, Jun Li, Yijin Zhang, and Yuwen Qian. Distributed vehicle tracking in wireless sensor network: A fully decentralized multiagent reinforcement learning approach. IEEE Sensors Letters, 5(1):1-4, 2020.
Michael L Littman. Markov games as framework for multi-agent reinforcement learning. In International Conference on Machine Learning, pages 157-163. 1994.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-ooppetivee environments. In Advances in Neural Information Processing Systems, pages 6379-6339, 2017.
Rahul Mazumder, Trevor Hastie, and Robert Tibshirani. Spectral regularization algorithms for learning large incomplete matrices. Journal of Machine Learning Research, 11(80):227-22322 2010.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, $1 518(7540):599-533 2015.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. In AAAI Conference on Artificial Intelligence, pages 1495-1502, 2018
Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate Q-value functions for decentralized POMDPs. Journal of Artificial Intelligence Research, 32:289-353, 2008
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmark- ing multi-agent deep reinforcement learning algorithms in cooperative tasks. arXiv preprint arXiv:2006.017 07769, 2021.
10

---

Under review as a conference paper at ICLR 2022
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shi- mon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference Machine Learning, pages 4295-4304, 2018.
Donald B Rubin. Multiple Imputation for Nonresponse in Surveys, volume 81. John Wiley & Sons, 2004.
Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung. Philip HS Torr, Jakob N Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In International Conference on Autonomous Agents and MultiAgent Systems, 2019.
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):34-359, 2017.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Hostallero, and Yung Yi. QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. International Conference on Machine Learning, pages 5896, 2019.
Daniel Stekhoven and Peter Bühlmann. MissForest non parametric missing value imputation for mixed-type data. Bioinformatics, 28(1): 2012.
Peter Sunehag, Guy Lever, Audrunas Gruslys. Wojciech Marian Czarnecki, Vinícius Flores Zam- baldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel 7. Leibo, Karl Tuyls, al. Value- decomposition networks for cooperative multi-agent learning based on team reward. In Interna- tional Conference on Autonomous Agents and MultiAgent Systems, pages 2085-2087, 2018.
Richard Sutton and Andrew G Barto. Reinforcement learning: introduction. 2011.
Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12(4))c0172395, 2017.
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In International Conference on Machine Learning, pages 330-337, 1993.
Bhekisipho Twala. Robot execution failure prediction using incomplete data. IEEE International Conference on Robotics and Biomimetics, pages 1513-1523, 2009.
Jun Yan, Bo Tang. and Haibo He. Detection of false data attacks in smart grid with supervised learning. International Joint Conference on Neural Networks, pages 1395-1402, 2016.
Jinsung Yoon, James Jordon, and Mihaela Schaar. GAIN: Missing data imputation using generative adversarial nets. In International Conference on Machine Learning, pages 5689-5698, 2018.

---

Under review as a conference paper at ICLR 2022
A HYPERPARAMETERS
We evaluate the performance the algorithm across 10 runs with different random seeds. We use grid search to fine-tune the learning rate and the number of hidden nodes of MADDPG and DDPG. For GAIN, we use the generator and the discriminator consisting of 2 hidden layers with 128 nodes, where ReLU activation used except for the last layers that sigmoid. We use Adam optimizer with learning rate of 0.001 for the training of GAIN. The hint probability Ph = 0.5 and &g 100 are used, and GAIN is trained every 5 episodes. The replay buffer for GAIN is initialized after each training.
We use MADDPG for IA-MARL. Each agent has the value function and the policy consisting of 2 hidden layers with 64 nodes. We use ReLU for activations except for the last layer of value function that does not use activation and the last layer of policy that uses tanh activation. We use Adam optimizer for all agents with the learning rate of 0.01 and use target networks with r 0.01. The length of the replay buffer is 10 MADDPG updates every 100 samples and collects 1024 samples before making initial update. Since the mask-based update makes the number of training data for IA-MARL smaller, we update IA-MARL every 100/(1 Pm)² samples and collects 1024/(1- pm)2 pm samples before making initial update. Accordingly, the target network parameters are less frequently updated in IA-MARL.
B ADDITIONAL EXPERIMENTAL RESULTS
In this Appendix, to show the effectiveness of IA-MARL in terms of the number of training data discussed in Section 4.2, show the training curves of MADDPG with missing training data and compare them with IA-MARL. Then we show the performance IA-MARL with different imputation methods. Furthermore, we evaluate the computational time of IA-MARL.
-40
-40
average rewards
average rewards
-80
-80
Pr
Ps:
P-
MADDPG, Ps
IA-MARL, Pm = 0.1, Mpr
100
I4-MARL Pm 0.2, 4
-120
DDPG
-120
DDPG
2000
4000 episodes
6000
8000
10000
2000
4000
6000
8000
10000
episodes
(b)
-60
-80
average rewards
average rewards
-100
-100
MADDPG,
MADDPG, Pm
MADDPG, 0.1
MADDPG, 0.2
140
IA-MARL, Pm Mp.. 10
1140
JA-MRLL p., 0.2, Nor 10"
DDPG
DDPG
-160
-160
30000
38000 episodes
episodes
(c)
(d)
Figure 5: (a)-b) Speaker-Listenes environment with a pair of agents. (c)-(d) Speaker- Listener environment with two pairs of agents.
12

---

Under review as a conference paper at ICLR 2022
Figure shows the performance of MADDPG with different Pm in Speaker-Listence environment with 7 2 and n = 4. We update MADDPG every Np 100/ =I Pm, )mm steps since the number of data for the training decreases due to the data missing. When the training data is missed,
except for Fig. 5a MADDPG has lower performance than IA-MARI. due to the small number of
training data E.
[Dpl |]
in equation 17 Note that the number of agents becomes larger, MADDPG
is more slowly trained, while the number of training data of IA-MARL is not affected by the number of agents.
average rewards
average rewards
-100
-100
-150
-150
MADOPG
MADDPG.
-20
DOPE
-20
DDPS
(-MARUGGHIN) 0.1
VAMARLL/AANN)
U.HARUranddmi,, Pae 0.1
W.M.RuUraddomi, 0
-250
Stripe-U--MALLICANNI =
-250
Simple-A-WAALLIAAIII Pr
Simplle (MMMMLIeoMM))))))) 0.1
Simple-la-WsLLLaancomm Pa 02
300
300
2000
4000 episodes
6000
8000
10000
2000
4000
6000
8000
10000
episodes
(a)
(b)
-50
-50
average rewards
average rewards
-100
-100
-150
-150
MADOPS, Ps
MADDPG, Pn
200
-200
WeMARLIGAIN), #.1
IA-MARLIGAIN,, Pa =01
-250
Simple-o-MARLICAIIN. 0.1
-250
Simple-9-MMALLIAANN, de 0.2
Sympteijo-M/ALLunmees token), Ps.
Stimpeess-KOORLLunbbs tokeni, Pay
300
300
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
episodes
episodes
Figure 6: IA-MARL and Simple -A-Ml--IA-MALL- with (a)-(b) random imputation and (c)-(d) unobserved token for the missing training data.
Figure 6 shows the performance of IA-MARL and Simple-AA-MARL with different imputation methods. In Figs. 6a and 6bb we replace the missing training data with uniform random variables, i.c., 0g, U(mino,, (oi.e), ; ma, (oi,))), where Ω is the set of all o,, In Figs. 6c and 6d replace the missing training data with the unobserved token -1, i.e., dg = -1. Note that for the
both IA-MARL and Simple-IA- IA-MARL, the GAIN shows the better performance than the random
imputation and the use of unobserved token for the missing training data.
Tag (n=5)
lag (n:4)
GAIN Training
Imputation
Listener
MARL Training
Sampling
(seconds)
Figure 7: Computation time of IA-MARL across 1000 episodes.
Figure 7 shows the computation time of IA-MARL across 1000 episodes when the numbers of agents 2, and 5, respectively. We use Intel i7-9700K CPU and NVIDIA GeForce RTX 2080Ti GPU
13

---

Under review as a conference paper at ICLR 2022
for the evaluation. The time for sampling and MARL training takes major portion compared to the time for the imputation and GAIN training. Note that as the number agents increases, the time for sampling and MARL training linearly increases since cach agent has the value function and the policy that should be trained. On the other hand, GAIN has two networks, i.e., the generator and the discriminator. Therefore, the time for imputation and GAIN training slowly increases with the number of agents compared with the time for sampling and MARL training.
14


Reviews:
[{"review_id": "xdqyiN_h3IE", "reviewer": "Reviewer_oFfr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Clear writing in the Introduction. The intro clarifies some of the important issues that inspired in this paper, the motivation, possible solutions, etc.\n2. The problem of MARL under missing data is interesting.\n3. The experiments are conducted under widely-used MARL benchmark environments.\n\n\nWeaknesses:\n1. Unconvincing experiments. This paper has abundant experiments, from different percentages of missing data to different environmental settings. However, I'm curious to see how the MADDPG without imputation works against with imputation. Maybe the  MADDPG without imputation itself could work well enough, and there is no need to impute. Since the missing data are randomly set, there are going to be cases where there is no missing data across all agents at that time. The authors should add this experiment, also with different missing rates, to convince the readers that imputation does work.\n\n2. The generality of the proposed method to a bigger domain. The imputation method used by this paper is GAIN, which is originally focused on vectorized data. This paper proposed to use GAIN to impute the trajectory, which includes states, actions, and rewards. The relationships between states/actions/rewards themselves could be complicated enough to model, and I wonder how this imputation could handle the relationships. What's more, when the state input is images, like in some settings in SMAC, generating an image would be difficult. Therefore, I'm concerned with the proposed method could only be applied when the environment setting is too simple to match the arguments mentioned in the intro: the communication failure, hardware limit, and security attacks in in wireless sensoror other real-world applications.\n", "summary_of_the_paper": "This paper proposes an imputation-assisted multi-agent reinforcement learning (IA-MARL) method under the problem setting of MARL, where the training data of each agent can be randomly missed with a certain probability. Specifically, IA-MARL uses a generative adversarial imputation network to impute the missing data, and train the MARL method on the imputed data.", "summary_of_the_review": "Based on the weaknesses mentioned in this paper, I tend to rate this paper as \"marginally below the acceptance threshold\". The main concerns are located in the experimental results and the complexity of the imputation problem itself. But I'm open to change my score after reading the rebuttal from the authors. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qKY0caTQSdW", "reviewer": "Reviewer_XnkH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strength of the IA-MARL paper lies in its problem specificity and the natural alignment between agent multiplicity in MARL research and the motivation for imputation: filling in the blanks. The flow and structure make the paper easy to read, and the imputation approach appears convincing and effective and poses some intriguing discussion points, which in my view could also constitute the paper's weaknesses. I would appreciate if the authors could include some of these discussion points in the paper, where they see fit.\n\nQ1. What would you say is the strongest motivation for employing imputation techniques in the multi-agent case? Do missing training data problems occur more frequently or cause more severe damage in the multi-agent case?\n\nQ2. In the single-agent case, if the imputation could be applied across the time axis (and not rely on the observations of other agents because there would be none), what benefits would you foresee? As a follow-up, how much would you say does IA-MARL take advantage of the information across the time axis or across the agent axis? In other words, when filling in the blanks for agent i, does IA-MARL learn more from the observation history of that agent, or from the current observations of other agents?\n\nQ3. As the authors imply, it is becoming more important to take into account real-world limitations when deploying MARL (e.g., communication failure). Could you discuss some relevance to a 2019 ICLR paper called SchedNet by Kim et al.? (which, to the best of my knowledge, is one of the earliest MARL works addressing real-life communications constraints.\n\nQ4. If time allows for it, it would be valuable to observe how IA-MARL stacks up against some of the more recent MARL methods of the authors' choice. I understand that none of them were designed to tackle the missing training data problem, but it may still be worthwhile to assess how they compete against each other in the missing training data setting: some works that come to my mind include: Deep Coordination Graphs, MAVEN, Bayesian Action Decoders, Stable Opponent Shaping, and QPLEX.\n\nQ5. I think the presented solution works well, but I am not entirely persuaded about how severe the missing training data problem is, possibly because I am not a security person. However, I do think that certain additions might strengthen the problem and the motivation by a large extent. For instance, Q5-a wouldn't you say that a poorly exploring team of agents is encountering a missing training data problem? (in the sense that all agents' training data is missing after time t). This case ties in with Q2; if IA-MARL learns across the time axis, then it could also carry out imputation in future time indices? Now, Q5-b, how would IA-MARL perform in situations where agent homogeneity does not hold? In the weak heterogeneity case, agents may have the same observation horizon (e.g., 3x3 grid around them) but some of them might have higher chance of missing data (e.g., due to poor comms link quality). In the strong heterogeneity case, agents could have different missing data ratios as well as different observation horizon, different observation frequency (e.g., some report at 1Hz and some others at 2Hz), and/or all of the aforementioned properties could vary across time (e.g., as comms link quality may also vary in time in e.g., FANETs). How would the authors find these kinds of factors with regard to the missing training data problem? (e.g., are they relevant/motivating or irrelevant?) I think these cases could help strengthen the missing training data problem the authors are trying to build; I would love to hear from the authors.", "summary_of_the_paper": "Authors present a novel and effective method leveraging a generative adversarial approach to a specific MARL problem with regard to missing training data. The approach presented treats the missing data as targets of imputation, loosely similar to that of inpainting problems in computer vision, where missing pixels are \"filled in\". Results show that IA-MARL outperforms one baseline (MADDPG) that has not been originally devised for missing training data.", "summary_of_the_review": "IA-MARL solution works well, but the missing training data problem does not look scary, as presented in the paper. I would gladly buy almost all of the claims presented, except the motivation behind the problem itself. To be specific, I will up my score if I could learn more from the authors about Q5 and at least one from Q1~Q4.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_xW7CTbmn8z", "reviewer": "Reviewer_SVEY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n- Simple, intuitive method that seems fairly effective in simple toy settings. \n\nWeaknesses:\n- Experimental results: \n1) Missing ablations and baselines: For every graph I would expect a comparison to state-of-the-art decentralized methods, such as independent PPO that only train on those transitions that are available for a given agent (i.e. not missing data). There is also a missing trivial baseline which simply inputs a \"not observed token\" (e.g. -1) for unobserved data and otherwise does standard CTDE. The current \"random imputation baseline\" samples from the entire range which seems like a strawman.\n2) No large scale experiments: The authors argue that SMAC can't be used as a testbed since MAPPO does poorly there. However, they also state that \"other actor-critic RL methods including policy gradient are also applicable for IA-MARL\". Multi-agent PPO is known to do well on SMAC, so why not test there? It's also unclear why the method should only be restricted actor-critic variants. What prevents the method from being used with e.g. VDM or QMIX?\n3) Clarity: It would be great to explain what the shading is in the plot. I assume this is standard error of the mean? The results also seem really noisy and somewhat hard to interpret.  \"we use DDPG for the prey and MADDPG for predators\": This statement is confusing. The plots for the tag environment contain MADDPG lines for both the prey and predators. Is this supposed to mean that each method is evaluated _against_ MADDPG predators and DDPG preys?\n\n- Conceptual:\nThe fact that masking is needed to make the method work is a red flag. It clearly illustrates that the imputation of the data does not actually recover the correct values. In principle, under centralized training, having wrong values for _the other_ agents can also be detrimental to learning, which would render the current masking ineffective. The fact that this doesn't happen in these problem settings seems like an accident and it would be great to explain any assumptions that make this possible.\nA good workaround for all of this might be to simply feed the imputation mask along with the confidence of the discriminator to both the agents and the value function. That way they can learn to utilise the data appropriately.\n", "summary_of_the_paper": "This paper develops an imputation method for missing data in multi-agent settings. The main idea is to train a GAN based generator that imputes missing data in the the centralised learning with decentralized execution (CTDE) setting.", "summary_of_the_review": "The method currently seems fairly incremental and it's unclear to me how broadly applicable it is. This is both due to the limited /noisy experimental evaluation and the missing discussion regarding the assumptions that went into the method and limitations. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0ihTcyJ98u", "reviewer": "Reviewer_vfgR", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Writing\nThe submission is poorly written, to say the least. I've quoted a couple of sentences, each with numerous problems, below. This is not meant as an exhaustive list requiring change but rather as representative of the quality of the writing found in the paper. I did not have to look hard to find examples -- these are the first sentence of the abstract and the last sentence of the first paragraph of the introduction, respectively.\n\n> Recently, multi-agent reinforcement learning (MARL) adopts the centralized training with decentralized execution (CTDE) framework that trains agents using the data from all agents at a centralized server while each agent takes an action from its observation.\n\n> Hence, the multi-agent reinforcement learning (MARL) that operates in multi-agent domain has been introduced and now becomes one of the most active and challenging RL research area.\n\n### Centralized vs Decentralized MARL\nThe submission does not accurately characterize the disadvantages of decentralized training (see below). Using a centralized value function during training neither resolves non-stationarity nor partial observability.\n\n> In MARL, the decentralized approach has been used to train each agent based on its trajectory (Tan, 1993). However, it often shows unstable and low performance due to non-stationary environment and partially observable information (Tan, 1993; Foerster et al., 2017) that inherits from the decentralization. Specifically, as the agents evolve their policies independently, the environment becomes non-stationary, which unstabilizes training at each agent. In addition, the agent may not observe the information of other agents, which causes low performance in the cooperative or competitive environment\n\nContrary to the tone of the submission, centralized value functions are not strictly beneficial. See Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning (2021).\n\n### Problem Setting\nI don't find the problem setting suggested by the submission to be well motivated. Could the submission give an application setting  that is currently addressed by decentralized MARL algorithms in which centralized MARL algorithms are inapplicable because of missing data? I think having such an example would help add credibility to the proposed problem setting.\n\n### Proposed Methodology\nThe proposed methodology is fairly simple (simply combining an imputation method with some masking). However, I do not think that that is a problem, so long as the problem setting is aptly justified.\n\n### Experiments\nThe submission shows some evidence that IA-MARL can outperform independent DDPG. However, for IA-MARL to merit the implementation trouble, it needs to offer better performance than a host of (non-centralized) MARL algorithms (e.g., independent PPO, independent R2D2 (minus the distributed part)), not just DDPG. I also concur with reviewer SVEY regarding the importance of of the baseline that inputs an \"unobserved token\" for unobserved data. \n\nThe submission states that it does not perform experiments on SMAC because MADDPG does not perform well in SMAC. I find this reasoning somewhat lacking. If IA-MARL is MADDPG-specific (which it isn't) and MADDPG does not perform well in some settings, that would be a significant downside for IA-MARL. But given that IA-MARL is not MADDPG-specific, and (as also noted by reviewer SVEY), MAPPO performs well on SMAC, its not clear why SMAC isn't an appropriate testbed.\n\nHyperparameter tuning can make a big difference in the performance of MARL algorithms. The appendix has some information about what hyperparameters the submission used but appears to be lacking information about how each algorithm examined in the submission was tuned. This information is especially important (in my opinion at least) because the submission is examining a novel problem setting in which there do not exist externally established performance metrics against which to compare.\n\n### Question\n\nIt is common practice in the cooperative MARL community to share parameters between agents (even when the agents are not using a centralized value function). Is this allowed in the problem setting proposed by the submission?", "summary_of_the_paper": "The submission proposes a cooperative MARL problem settings in which the observation-action-reward tuples generated during training are unavailable with some non-zero probability. The submission suggests addressing this problem setting by first imputing the missing training data and what they call a mask based update. The submission presents experiments for multi-agent particle environments.", "summary_of_the_review": "Overall, I think the submission would need to\n\n- Do more to justify the idea that this is a problem setting in need of attention\n- Address the deficiencies in writing quality\n- Add additional relevant baselines / experiments\n- Add additional details regarding hyperparameter tuning\n\nto merit acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper tackles the problem of missing data in centralized training multi-agent RL approaches. The authors propose 1) using generative adversarial imputation networks for imputing missing data and 2) discarding training data where data from multiple consecutive timesteps is missing.

Reviewers agreed that the problem of missing data in multi-agent RL is interesting. At the same time, several reviewers shared two main concerns about the experimental evaluation:
* The lack of comparisons to baselines other than MADDPG, especially decentralized critic approaches.
* The lack of experiments on non-toy domains such as SMAC.

The author response did not sufficiently address these concerns leaving the reviewers in agreement that the paper should not be accepted without these additional experiments.
Decision: Reject