Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Authors present a novel and effective method leveraging a generative adversarial approach to a specific MARL problem with regard to missing training data. The approach presented treats the missing data as targets of imputation, loosely similar to that of inpainting problems in computer vision, where missing pixels are "filled in". Results show that IA-MARL outperforms one baseline (MADDPG) that has not been originally devised for missing training data.
main_review: The strength of the IA-MARL paper lies in its problem specificity and the natural alignment between agent multiplicity in MARL research and the motivation for imputation: filling in the blanks. The flow and structure make the paper easy to read, and the imputation approach appears convincing and effective and poses some intriguing discussion points, which in my view could also constitute the paper's weaknesses. I would appreciate if the authors could include some of these discussion points in the paper, where they see fit.

Q1. What would you say is the strongest motivation for employing imputation techniques in the multi-agent case? Do missing training data problems occur more frequently or cause more severe damage in the multi-agent case?

Q2. In the single-agent case, if the imputation could be applied across the time axis (and not rely on the observations of other agents because there would be none), what benefits would you foresee? As a follow-up, how much would you say does IA-MARL take advantage of the information across the time axis or across the agent axis? In other words, when filling in the blanks for agent i, does IA-MARL learn more from the observation history of that agent, or from the current observations of other agents?

Q3. As the authors imply, it is becoming more important to take into account real-world limitations when deploying MARL (e.g., communication failure). Could you discuss some relevance to a 2019 ICLR paper called SchedNet by Kim et al.? (which, to the best of my knowledge, is one of the earliest MARL works addressing real-life communications constraints.

Q4. If time allows for it, it would be valuable to observe how IA-MARL stacks up against some of the more recent MARL methods of the authors' choice. I understand that none of them were designed to tackle the missing training data problem, but it may still be worthwhile to assess how they compete against each other in the missing training data setting: some works that come to my mind include: Deep Coordination Graphs, MAVEN, Bayesian Action Decoders, Stable Opponent Shaping, and QPLEX.

Q5. I think the presented solution works well, but I am not entirely persuaded about how severe the missing training data problem is, possibly because I am not a security person. However, I do think that certain additions might strengthen the problem and the motivation by a large extent. For instance, Q5-a wouldn't you say that a poorly exploring team of agents is encountering a missing training data problem? (in the sense that all agents' training data is missing after time t). This case ties in with Q2; if IA-MARL learns across the time axis, then it could also carry out imputation in future time indices? Now, Q5-b, how would IA-MARL perform in situations where agent homogeneity does not hold? In the weak heterogeneity case, agents may have the same observation horizon (e.g., 3x3 grid around them) but some of them might have higher chance of missing data (e.g., due to poor comms link quality). In the strong heterogeneity case, agents could have different missing data ratios as well as different observation horizon, different observation frequency (e.g., some report at 1Hz and some others at 2Hz), and/or all of the aforementioned properties could vary across time (e.g., as comms link quality may also vary in time in e.g., FANETs). How would the authors find these kinds of factors with regard to the missing training data problem? (e.g., are they relevant/motivating or irrelevant?) I think these cases could help strengthen the missing training data problem the authors are trying to build; I would love to hear from the authors.
summary_of_the_review: IA-MARL solution works well, but the missing training data problem does not look scary, as presented in the paper. I would gladly buy almost all of the claims presented, except the motivation behind the problem itself. To be specific, I will up my score if I could learn more from the authors about Q5 and at least one from Q1~Q4.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an imputation-assisted multi-agent reinforcement learning (IA-MARL) method under the problem setting of MARL, where the training data of each agent can be randomly missed with a certain probability. Specifically, IA-MARL uses a generative adversarial imputation network to impute the missing data, and train the MARL method on the imputed data.
main_review: Strengths:
1. Clear writing in the Introduction. The intro clarifies some of the important issues that inspired in this paper, the motivation, possible solutions, etc.
2. The problem of MARL under missing data is interesting.
3. The experiments are conducted under widely-used MARL benchmark environments.


Weaknesses:
1. Unconvincing experiments. This paper has abundant experiments, from different percentages of missing data to different environmental settings. However, I'm curious to see how the MADDPG without imputation works against with imputation. Maybe the  MADDPG without imputation itself could work well enough, and there is no need to impute. Since the missing data are randomly set, there are going to be cases where there is no missing data across all agents at that time. The authors should add this experiment, also with different missing rates, to convince the readers that imputation does work.

2. The generality of the proposed method to a bigger domain. The imputation method used by this paper is GAIN, which is originally focused on vectorized data. This paper proposed to use GAIN to impute the trajectory, which includes states, actions, and rewards. The relationships between states/actions/rewards themselves could be complicated enough to model, and I wonder how this imputation could handle the relationships. What's more, when the state input is images, like in some settings in SMAC, generating an image would be difficult. Therefore, I'm concerned with the proposed method could only be applied when the environment setting is too simple to match the arguments mentioned in the intro: the communication failure, hardware limit, and security attacks in in wireless sensoror other real-world applications.

summary_of_the_review: Based on the weaknesses mentioned in this paper, I tend to rate this paper as "marginally below the acceptance threshold". The main concerns are located in the experimental results and the complexity of the imputation problem itself. But I'm open to change my score after reading the rebuttal from the authors.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper develops an imputation method for missing data in multi-agent settings. The main idea is to train a GAN based generator that imputes missing data in the the centralised learning with decentralized execution (CTDE) setting.
main_review: Strengths: 
- Simple, intuitive method that seems fairly effective in simple toy settings. 

Weaknesses:
- Experimental results: 
1) Missing ablations and baselines: For every graph I would expect a comparison to state-of-the-art decentralized methods, such as independent PPO that only train on those transitions that are available for a given agent (i.e. not missing data). There is also a missing trivial baseline which simply inputs a "not observed token" (e.g. -1) for unobserved data and otherwise does standard CTDE. The current "random imputation baseline" samples from the entire range which seems like a strawman.
2) No large scale experiments: The authors argue that SMAC can't be used as a testbed since MAPPO does poorly there. However, they also state that "other actor-critic RL methods including policy gradient are also applicable for IA-MARL". Multi-agent PPO is known to do well on SMAC, so why not test there? It's also unclear why the method should only be restricted actor-critic variants. What prevents the method from being used with e.g. VDM or QMIX?
3) Clarity: It would be great to explain what the shading is in the plot. I assume this is standard error of the mean? The results also seem really noisy and somewhat hard to interpret.  "we use DDPG for the prey and MADDPG for predators": This statement is confusing. The plots for the tag environment contain MADDPG lines for both the prey and predators. Is this supposed to mean that each method is evaluated _against_ MADDPG predators and DDPG preys?

- Conceptual:
The fact that masking is needed to make the method work is a red flag. It clearly illustrates that the imputation of the data does not actually recover the correct values. In principle, under centralized training, having wrong values for _the other_ agents can also be detrimental to learning, which would render the current masking ineffective. The fact that this doesn't happen in these problem settings seems like an accident and it would be great to explain any assumptions that make this possible.
A good workaround for all of this might be to simply feed the imputation mask along with the confidence of the discriminator to both the agents and the value function. That way they can learn to utilise the data appropriately.

summary_of_the_review: The method currently seems fairly incremental and it's unclear to me how broadly applicable it is. This is both due to the limited /noisy experimental evaluation and the missing discussion regarding the assumptions that went into the method and limitations.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The submission proposes a cooperative MARL problem settings in which the observation-action-reward tuples generated during training are unavailable with some non-zero probability. The submission suggests addressing this problem setting by first imputing the missing training data and what they call a mask based update. The submission presents experiments for multi-agent particle environments.
main_review: ### Writing
The submission is poorly written, to say the least. I've quoted a couple of sentences, each with numerous problems, below. This is not meant as an exhaustive list requiring change but rather as representative of the quality of the writing found in the paper. I did not have to look hard to find examples -- these are the first sentence of the abstract and the last sentence of the first paragraph of the introduction, respectively.

> Recently, multi-agent reinforcement learning (MARL) adopts the centralized training with decentralized execution (CTDE) framework that trains agents using the data from all agents at a centralized server while each agent takes an action from its observation.

> Hence, the multi-agent reinforcement learning (MARL) that operates in multi-agent domain has been introduced and now becomes one of the most active and challenging RL research area.

### Centralized vs Decentralized MARL
The submission does not accurately characterize the disadvantages of decentralized training (see below). Using a centralized value function during training neither resolves non-stationarity nor partial observability.

> In MARL, the decentralized approach has been used to train each agent based on its trajectory (Tan, 1993). However, it often shows unstable and low performance due to non-stationary environment and partially observable information (Tan, 1993; Foerster et al., 2017) that inherits from the decentralization. Specifically, as the agents evolve their policies independently, the environment becomes non-stationary, which unstabilizes training at each agent. In addition, the agent may not observe the information of other agents, which causes low performance in the cooperative or competitive environment

Contrary to the tone of the submission, centralized value functions are not strictly beneficial. See Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning (2021).

### Problem Setting
I don't find the problem setting suggested by the submission to be well motivated. Could the submission give an application setting  that is currently addressed by decentralized MARL algorithms in which centralized MARL algorithms are inapplicable because of missing data? I think having such an example would help add credibility to the proposed problem setting.

### Proposed Methodology
The proposed methodology is fairly simple (simply combining an imputation method with some masking). However, I do not think that that is a problem, so long as the problem setting is aptly justified.

### Experiments
The submission shows some evidence that IA-MARL can outperform independent DDPG. However, for IA-MARL to merit the implementation trouble, it needs to offer better performance than a host of (non-centralized) MARL algorithms (e.g., independent PPO, independent R2D2 (minus the distributed part)), not just DDPG. I also concur with reviewer SVEY regarding the importance of of the baseline that inputs an "unobserved token" for unobserved data. 

The submission states that it does not perform experiments on SMAC because MADDPG does not perform well in SMAC. I find this reasoning somewhat lacking. If IA-MARL is MADDPG-specific (which it isn't) and MADDPG does not perform well in some settings, that would be a significant downside for IA-MARL. But given that IA-MARL is not MADDPG-specific, and (as also noted by reviewer SVEY), MAPPO performs well on SMAC, its not clear why SMAC isn't an appropriate testbed.

Hyperparameter tuning can make a big difference in the performance of MARL algorithms. The appendix has some information about what hyperparameters the submission used but appears to be lacking information about how each algorithm examined in the submission was tuned. This information is especially important (in my opinion at least) because the submission is examining a novel problem setting in which there do not exist externally established performance metrics against which to compare.

### Question

It is common practice in the cooperative MARL community to share parameters between agents (even when the agents are not using a centralized value function). Is this allowed in the problem setting proposed by the submission?
summary_of_the_review: Overall, I think the submission would need to

- Do more to justify the idea that this is a problem setting in need of attention
- Address the deficiencies in writing quality
- Add additional relevant baselines / experiments
- Add additional details regarding hyperparameter tuning

to merit acceptance.
review_title: null

Meta-review: The paper tackles the problem of missing data in centralized training multi-agent RL approaches. The authors propose 1) using generative adversarial imputation networks for imputing missing data and 2) discarding training data where data from multiple consecutive timesteps is missing.

Reviewers agreed that the problem of missing data in multi-agent RL is interesting. At the same time, several reviewers shared two main concerns about the experimental evaluation:
* The lack of comparisons to baselines other than MADDPG, especially decentralized critic approaches.
* The lack of experiments on non-toy domains such as SMAC.

The author response did not sufficiently address these concerns leaving the reviewers in agreement that the paper should not be accepted without these additional experiments.
Decision: Reject