Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new value factorization method for cooperative MARL. The technique brings out an alternate way of value factorization (to WQMIX and QMIX) when the target values are non-monotonic and stochastic. In particular, prior approaches is the inability to recover the optimal Qtot value when the joint value function is non-monotonic and stochastic. A popular matrix game example variant shows that the previous approaches frequently overestimate or underestimate joint Q values in stochastic and non-monotonic settings. An alternate shaping function is introduced to generate the shaped joint action-value that can consider the stochasticity in the reward and state function and place appropriate weight on the different joint actions. To this end, a neural network is trained to predict standard deviation and error in the reward and the embedding of the next state. Another neural network is trained to estimate the best action value for each agent, assuming the agent gets the cooperation of other agents. Finally, based on the prediction error and the best action values, the shaping function generates monotonic target joint action-values for QMIX.
strength_and_weaknesses: Strengths
•	Incorporates stochasticity in reward and state representation
•	Good empirical performance
•	Limitations of prior works is explained through theory and the example of matrix game

Weaknesses
•	The overall approach is a bit of a kitchen sink with many hyperparameters to be tuned and models to be trained. QMIX alone is already known for being challenging on this front with larger number of agents.  See, e.g., the below reference.
•	Goal of estimating the stochasticity of rewards and next states seems similar in spirit to distributional RL, but there is no comparison to related work from this literature.
•	While the experiments include some ablations, the role of various parts of the overall approach is not fully explored.  For example, there are two approaches used in combination to detect stochasticity (reward predictor and state predictor).  What if you only include one?  Or what happens if you use different rules about when to apply or not apply the change of target than that in Equation (7) since at the very least it has two different cases.

@article{avalos2021local,
  title={Local Advantage Networks for Cooperative Multi-Agent Reinforcement Learning},
  author={Avalos, Rapha{\"e}l and Reymond, Mathieu and Now{\'e}, Ann and Roijers, Diederik M},
  journal={arXiv preprint arXiv:2112.12458},
  year={2021}
}

clarity,_quality,_novelty_and_reproducibility: The use of random embedding for estimating stochasticity seems similar to some work on exploration such as the following (although with a different goal).  Is there a relation?

@article{burda2018exploration,
  title={Exploration by random network distillation},
  author={Burda, Yuri and Edwards, Harrison and Storkey, Amos and Klimov, Oleg},
  journal={arXiv preprint arXiv:1810.12894},
  year={2018}
}

The first sentence of 4.2 is confusing to me.  I thought Q^{mvf} is intended to be the set of things that QMIX can in principle learn.  But the phrasing suggests that there are things in it that QMIX cannot represent.  Is there a typo here or something that needs to be better explained?

The reason of predicting each agent’s best action value function is not well explained in the main text and required studying the proof of Theorem 2 for me to understand.  My intuition is the goal is quantify the suboptimality of actions because, by definition, the the join action is suboptimal (and the optimal action is unique) then for every agent not playing their part of the optimal action their q_a will be less than the optimal Q value.  

The statement of Theorem 2 seems to be missing something.  It contains a “such that,” with no indication what the assumption is.  From the proof I’m guessing it is that the optimal policy is unique and the discussion of this issue in the proof should be discussion and not actually part of the proof?  Relatedly, I don’t know where there is a quantification over joint actions u in the theorem statement.

The intuition for the decision not to change the target if it is highly stochastic is not clear to me. (Discussion before (8))

summary_of_the_review: A paper with strong empirical performance but room for improvement in the throughness and clarity of the exposition.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a reward shaping scheme for MARL towards fixing QMIX, an existing algorithm suffering from suboptimality in non-monotonic settings. The method uses target shaping by predicting reward and next state uncertainty. 
strength_and_weaknesses: 1. There are some important works missing in the related works which need to be discussed towards getting a proper context for learning in CTDE settings (instead of focusing only on value based methods) eg. : Tesseract [1] an actor-critic method natively supports decentralization despite environment stochasticity and non monotonicity, it also proves important representation capacity results. Similarly, there have been developments in model based methods [2] for addressing representational issues.

2. How does the proposed method deal with uncertainty arising from neural network approximation? this is a huge component of target noise in practise. There isn't enough discussion on this given that the proposed method relies on uncertainty estimates that too with limited number of environment steps.

3. What is meant by "convergence to the optimal QMIX for any non-monotonic and stochastic Q"? is it the best qmix learnable policy?

4. How exactly is eq 7 preventing a bad monotonic projection? There needs to be a guarantee in probabilisitic or worst case sense.

5. Why are stronger baselines not used for comparison? For eg. [3] has better performance on all the SMAC maps considered here. Overall this also affects significance of the work if strictly better methods already exist.

6. The paper claims to be aimed for stochastic domains but SMAC is not adequately stochastic.

7. The paper needs improvement in terms of writing and presentation, especially for conveying the overall problem and method. Currently, I do not find it easily readable.

Minors:
1. The results used from other works should be cited where they are used: eg. The line in introduction "However, QMIX can only represent values in the restricted monotonic space" should cite Maven, Mahajan et al 2019 and Qtran, Son et al 2019, there itself.
2. Eq 3 has been repeated twice.
3. Pg.5 ending, "objection of", did you mean "objective of"?

References:
1. Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learning, Mahajan et al, 2021
2. Model based Multi-agent Reinforcement Learning with Tensor Decompositions, Van Der Vaart et al 2021
3. Rode: Learning roles to decompose multi-agent tasks, Wang et al, 2021

clarity,_quality,_novelty_and_reproducibility: - Writing needs improvement as the paper is not very clear. Related works also needs to address MARL approaches more broadly.
- The paper tries to fix problems in an existing method.
- There is also need for better empirical evaluation on complex domains with actual stochasticity. The performance compared to existing methods like [3] is also not well. Some of the claims about representability achieved by reward shaping is not well supported with proofs.
- SMAC is an opensourced domain, the  code for the method and predator prey environment has not been provided.
summary_of_the_review: Interesting idea but needs improvements (see the 2 sections above).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The submission proposes a new CTDE algorithm called UTS. UTS is designed to have two properties that the submission calls policy invariance and full representational capacity. The submission benchmarks UTS in matrix games, predator prey, and SMAC.
strength_and_weaknesses: ### Clarity

I personally found the description of the problems that UTS is trying to solve somewhat obfuscated. Perhaps I am not the target audience and that is why I found it difficult to follow, but I think the submission could be clearer overall. To give a concrete example, I feel like I read through the whole intro without really understanding the main idea.

### Baselines

It seems like the submission is lacking some relevant baselines, such as MAPPO, IPPO, and HATRPO, given that it is claiming to "outperform state-of-the-art".

### Benchmarking

In deep RL, and perhaps especially deep MARL, there is historical precedent for bad empirical practices, such as reporting mean return over a small number of seeds. It is well documented that this practice leads to unreliable reporting of algorithmic strength -- see Deep Reinforcement Learning at the Edge of the Statistical Precipice (NeurIPS 2021). I think the submission ought report the alternative metrics advocated in the statistical precipice paper if it wants to make big claims like "outperforming state-of-the-art".

### Motivation

> In this section, we introduce a shaping function to generate the shaped joint action-value Qf(s, u) to replace the original Q(s, u). The shaping function should have the following properties: 1) policy invariance, i.e., not making the agent deviate from the true goal and keeping the optimal policy unchanged; 2) full representational capacity, i.e., all shaped joint action-values should belong to
a subset of Qmvf.

This is a really important sentence in the submission. Yet, it doesn't feel clear to me. What does "not making the agent deviate from the true goal" mean? Additionally, why are we saying that "shaped joint action-values should belong to a subset of Qmvf"? Isn't belonging to a subset of a set the same as belonging to the set itself?

### Dealing with stochasticity

The approach to determining whether stochasticity is present feels unsatisfactory to me. The state predictor may output zero even if there is stochasticity.

### Confusion

> However, these approaches suffer from instability arising from the non-stationarity of the environment induced by simultaneously
learning and exploring agents (Kuba et al., 2021). Centralised learning of joint action-values can solve these problems, but it is challenging to scale in real-world applications due to intractably large joint action space and communication constraints. 

Kuba et al. (2021) show that MAPPO-style approaches lack a monotonic improvement guarantee. Unless I am misunderstanding, UTS does not have such a guarantee either. Could the authors clarify this point?
clarity,_quality,_novelty_and_reproducibility: I personally found some aspects of the clarity to be lacking, as discussed above. I also raised some issues regarding the quality of the empirical approach (ie, dealing with stochasticity) and the quality of the baselines and benchmarking. To my knowledge, the approach is novel. The code is not available; I would guess that it would difficult to independently reproduce the results.
summary_of_the_review: The method feels a little hacky to me, but I think that's not the worst thing if it performs well. 

The submission makes big claims about its method's performance, such as "outperforming state-of-the-art". However, I do not feel that it lives up to these claims in the experiments, both because it is lacking in baselines and because it does not follow modern deep RL benchmarking practices.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper aims to address two shortcomings of the popular methods QMIX for and WQMIX for learning in Dec-POMDPs. 
Through simple toy example the authors show that these methods struggle both with non-monotonic and stochastic settings and propose a methods (UTS) that addresses these issues. 

Strength: 
I believe that there is real insight in this paper and that the analysis of failure cases / conceptual parts of the work are original and informative. 

Weakness: 
Some of the reviewers complained that the experimental evaluation was not thorough enough. This is partially something that the entire field of MARL suffers from but the authors should have at least compared to MAPPO in their original submission, rather than doing this only during the rebuttals.
The method it also rather complicated and it would be great to have a more in-depth ablation study of the different components. 

On balance I believe it would be preferable to submit an updated manuscript which addresses all of the issues raised by the reviewers to a future conference for a fresh and unbiased set of reviews. 
justification_for_why_not_higher_score: Some of the reviewers complained that the experimental evaluation was not thorough enough. This is partially something that the entire field of MARL suffers from but the authors should have at least compared to MAPPO in their original submission, rather than doing this only during the rebuttals.
The method it also rather complicated and it would be great to have a more in-depth ablation study of the different components. 

On balance I believe it would be preferable to submit an updated manuscript which addresses all of the issues raised by the reviewers to a future conference for a fresh and unbiased set of reviews.
Decision: Reject