Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes Model-Based Value Decomposition (MBVD), an implicit model-based multi-agent reinforcement learning (MARL) method that integrates the idea of model learning into value decomposition. The authors are enlightened by the ability of long-term vision of humans, which refers to humans’ consideration of future states after several interactions with the environment following their current strategies. Thus the authors let the agents cooperate by obtaining the aggregated latent states in the future. They elaborate on MBVD by discussing the framework, the flowchart, and the learning objectives of the algorithm. The MBVD method is tested in different domains, such as StarCraft II, Google Research Football, and Multi-Agent MuJoCo. The paper clarifies that MBVD exceeds the performance of several baselines in terms of sample efficiency. A derivation of the evidence lower bound (ELBO) and the settings of the environments are included in the appendix.
strengths_and_weaknesses: Strengths:

This work has good originality. The authors state that the study is the first attempt to apply the model-based reinforcement learning ideas to decentralized partially observable Markov decision process (Dec-POMDP) problems, which is true to the best of my knowledge.
The manuscript is clearly written. Important contents, such as the method itself and the results of the experiments, are presented in a well-organized way. Some more details like the experimental setup are placed in the appendix. 

The proof of breaking down the ELBO (Eq. 1) is given in the appendix, which is correct to me and improves the soundness of the submission. The manuscript proves itself to be a complete piece of work rather than a work in progress.

Weaknesses:

In the Related Work section, the authors used a long paragraph to talk about Single-Agent Model-based RL. However, it would be better if the majority of the discussion is on Multi-Agent Model-based RL because the authors’ research is exactly about MARL. There may be relatively fewer papers on multi-agent model-based RL, but the authors can discuss each of them in greater detail.

In the Experiments section, the performance of MBVD is not very impressive in relatively simpler scenarios of StarCraft II. It is hard to decide whether MBVD outperforms other baselines in every hard scenario because only representative scenarios are selected. Also, the ablation studies only show the results from the 2c_vs_64zg map. I am curious to see results from other scenarios for a better generalization.

questions: In lines 33-34, should the “decentralized partial observable Markov decision process” be “decentralized partially observable Markov decision process” instead?

In line 237, what is the definition of Detach(·)? I can only find the description of what this function can do. References can be added here, but are not necessary.

In lines 142 and 149, the authors mentioned that they used two VAEs as the prior and the posterior for the MBVD framework. Because there have been many variants of the VAE model since it was proposed by Kingma and Welling in 2014, I don’t know which one was used by the authors or if they made a new variant.   

From what I understand, when we apply Equation 2 to the experiments in Google Research Football and Multi-Agent MuJoCo, the last error (L_{FA}) is 0 because predictions of the feasible action set are only made in StarCraft II?

limitations: The authors specified the limitation of the paper is that the value of the rollout horizon was chosen manually. It would be nice if they can elaborate on the limitations, for example, by discussing how other researchers deal with the rollout horizon in their works. It does not seem likely that this paper will cause any potential negative social impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces the model-based value decomposition (MBVD) framework, which extends a value decomposition method (e.g., QMIX) with the imagination module. Specifically, the imagination module consists of two VAEs and outputs the rollout of imagined future states in the latent space ($\hat{s}^{\text{Rollout}}_{t}$). Then, this rollout information is concatenated with the current state ($s_t$) and becomes an input to the mixing network of QMIX. MBVD assumes that these imagined states contain helpful information for agents to evaluate the current state's value more accurately. Empirical results show that MBVD outperforms baselines in various domains of SMAC, Football, and multi-agent MuJoCo. 
strengths_and_weaknesses: **Strengths:**
1. The paper introduces a new framework of how learned models can be integrated into an existing value decomposition method. 
2. The evaluation is performed across complex domains, demonstrating the scalability of MBVD.

**Weaknesses:**
1. Because this paper's main contribution is related to the model-based aspects, it would be desirable to have comparisons against existing model-based multi-agent baselines, but there are no comparisons against model-based multi-agent baselines in the evaluation. 
2. MBVD uses the existing methods in Section 4.3 (e.g., VAE loss, QMIX loss), so there might not be a new contribution from the optimization perspective. However, as noted in Strength #1, how learned models can be combined in QMIX is new, and if this paper studied deeper on various ways to integrate learned models (other than concatenating the imagined states to the current state), this contribution could have been more significant.
questions: 1. In Section 4.2, the paper notes that the aggregated rollout state ($\hat{s}^{\text{Rollout}}_{t}$) is concatenated with the real global state ($s_t$). However, this paper assumes Dec-POMDP settings, so it is not possible to observe the real global state information. As such, it is confusing how the real global state information ($s_t$) can be obtained in MBVD.
2. After reading Section 5.4 (ablation studies), it is confusing to interpret the results for MBVD vs QMIX-RS. Because QMIX-RS receives the actual trajectories instead of predictions (which can be noisy approximations of true trajectories), I thought QMIX-RS would be the upper-bar performance of MBVD, but this is not true in Figure 6(a). Would it be possible to ask for a clarification?
3. Figure 7 is not straightforward to compare the real and imagined latent states. Possibly, it might be better to have 3 subfigures instead of 4 subfigures by overlaying the real latent state subfigure to the imagined latent state subfigures.
4. It is not specified how $p(u_t|z_t)$ is represented (e.g., $p(u_t|z_t)=\prod_a p(u^a_t|z^a_t)$) 
5. There is a typo in Section 2.2: "MBVD we proposed".
6. A citation is missing for Dec-POMDP in Section 3.1.
7. In Section 4.2, the (fixed) latest policies of all agents are used for the rollout. However, in practice in the future, they will have non-stationary policies due to their policy updates, so I wonder whether it will be helpful to consider their non-stationary policies in generating the rollout as a future direction (related works: Foerster et al., 2018; Kim et al., 2021). 

**References:**  
Foerster et al., DiCE: The Infinitely Differentiable Monte-Carlo Estimator, 2018.  
Kim et al., A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning, 2021.  
limitations: N/A
review_title: null

Meta-review: metareview: The reviewers appreciated the paper's originality in its combination of existing components, solid theoretical motivation, clear writing, and evaluation on complex problems. For these reasons, I recommend acceptance.
confidence: Certain
Decision: Accept