Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce DRIMA - a distributional CTDE multi-agent RL approach that separately learns to model return stochasticity arising from other agents vs the environment. They argue this is useful because for example it allows agent to be optimistic wrt teammates (who can prosocially adapt) but more risk-neutral wrt the environment (which does not adapt). They evaluate the effectiveness of their approach on the Starcraft MultiAgent Challenge (SMAC), and outperform several state-of-the-art baselines.
main_review: STRENGTHS

The motivation behind DRIMA is intuitive and interesting, and the implementation is non-trivial. The authors compare to a wide range of appropriate baselines on a challenging community benchmark, and achieve strong and convincing performance.

WEAKNESSES

The experiments in the paper jump straight to the SMAC domain, without testing their approach in a diagnostic illustrative environment. This made it difficult to understand some aspects of their model. For example, the authors find in Fig 6 that DRIMA seems to perform best when risk-seeking in terms of both agent-wise and environment-wise risk. This runs counter to the authors stated motivation of designing agents that can be optimistic wrt teammates but more risk-neutral wrt the environment. Unfortunately, the authors don't seem to find other tasks/environments for which different risk balances are optimal. This raises the question - is it always better for DRIMA agents to be risk-seeking in both regards? If so, why is it important to learn a factorizied representation of uncertainty anyway? A diagnostic illustrative environment would allow the authors to explore these questions, and would result in more clear scientific contribution of the paper.

Another example question of mine that went unanswered - presumably the reason to be risk-seeking wrt teammates and risk-neutral wrt the environment is that teammates are learning and so can be expected to prosocially adapt. For this reason, would DRIMA perform better than baselines if one teammate learnt slower than the other, or if their learning were reset part way through training? The intuition is that DRIMA might be better at helping lagging teammates "catch up" due to this optimism.

Also, although reviewers are instructed not to penalize papers for writing style, the current draft suffers from enough grammatical errors so as to seriously damage readability. Thus the paper would benefit from significant editorial review.

Lastly, the intuition that uncertainty and optimism over other agents can help facilitate cooperation was also discussed in Baker 2020 (https://arxiv.org/abs/2011.05373), which the authors might find interesting.
summary_of_the_review: As is, I found the results interesting and convincing enough to be just over the bar for acceptance. However, I am fairly uncertain in my assessment in part due to a lack of familiarity with the relevant distributional RL literature. In any case, I would feel more strongly about acceptance if the writing clarity were improved and additional diagnostic explanatory experiments were added, as outlined above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel multi-agent reinforcement learning algorithm that disentangles randomness/risk sources coming from (i) unobservable actions of cooperative agents and (ii) unobservable actions of enemy agents (environment stochasticity). The proposed method, DRIMA, uses Implicit Quantile Networks (IQN) to learn the joint action action-value distribution and controls the environment-wise risk level by changing its sampling distribution. The agent-wise risk level, on the other hand, is controlled through a hyperparameter in the loss function that softly ignores learning of the action-value function for non-optimal actions, and learns it accurately only for optimal actions in which agents are fully cooperative (similarly to previous approaches, e.g. Weighted QMIX). The proposed method shows improved performance compared to other state-of-the-art MARL algorithms in difficult StarCraft benchmark tasks.
main_review: Strengths:
- The paper is well-written and appropriately discusses the relevant literature of related MARL algorithms.
- The idea of disentangling agent-wise and environment-wise risks is novel and applying risk-sensitive reinforcement learning in multi-agent reinforcement learning is important to the field.
- The experiments in the difficult StarCraft tasks show an advantage of the proposed method in relation to other state-of-the-art MARL algorithms. The ablation study shows that environment-wise risk-seeking helps in tasks where more exploration is necessary.

Weaknesses:
- I missed a more deep discussion and comparison with DFAC (Sun et al., 2021) and other distributional MARL algorithms. Is the environment-wise risk control in both algorithms the same (that is, following IQN)?  It is not clear whether the only contribution of DRIMA is introducing the agent-wise risk level through the Weighted QMIX loss function. 
- The agent-wise risk level w_agt in the loss for non-optimal actions in Section 3.1 seems to only control whether it will follow the loss function from QTRAN (w_agt=1) or from Weighted QMIX. In the latter case, is the w_agr parameter playing the exact same role as the parameter alpha in the Weighted QMIX loss function? If that is the case, the only contribution here is the interpretation of this parameter as a risk level control.
- In the experiments it seems that agent-wise risk-seeking always shows higher performance (that is, following QTRAN loss function with w_agt=1). It is not clear what is the advantage of using different values for w_agt in terms of risk control.
- Although common in the deep MARL literature (especially considering the computationally expensive StarCraft tasks), the paper does not justify why four random seeds are sufficient to make their claims.

Furthermore, I have the following questions and criticisms:

- Section 2.1 is not very self-contained. I needed to read related works (e.g. QMIX) to understand why the decomposition of Eq. 1 can be applied. I suggest the authors improve the discussion on when it is reasonable to expect this condition to be true and its implications.
- In Eq. 3, “y is the true target action-value by unconstrained true action-value estimator Qjt”. Does this mean that y equals Qjt(tau,u), as denoted in Eq. 2? If so, I suggest the authors make it consistent between both equations.
- “Under this paradigm, value-based one has gained impressive attention,” 
It is not clear what “one” is referring to in this sentence.
 - “(a) agent-wise and (b) environment-wise, which do not emerge in the single-agent setting”
Environment-wise risk can also occur in the single-agent setting. E.g. a stochastic reward function can sometimes give higher or lower rewards to the agent.
- In the evaluation period, it is possible to choose actions with different risk levels by simply changing the value of w_agt. However, is it possible to control the environment-wise risk level after training? If I understand correctly, this is not possible since the individual agent networks do not depend on w_env.
- “DFAC is another distributional MARL framework with proposed mean-shape decomposition while employing IQN network for agent-wise utility function”
The meaning of mean-shape decomposition is not defined, I believe.
- “Furthermore, (s, u, r, s’) is a tuple of objects collected from consecutive time steps in the Markov decision process”
I believe the authors meant a  tuple of experience transitions. Is that the case? It is not clear what “objects” are, here.


======== UPDATE ========

I have increased my score to "weak accept" after the author's clarifications and novel additions to the draft. They significantly clarified many of my concerns for the theoretical justifications of the agent-wise and environment-wise "risk" parameters.
summary_of_the_review: The paper is well-written and the algorithm is well-justified. However, the main technical contributions seem to come from previous approaches (agent-wise risk level from Weighted Q-MIX and environment-wise risk level from DFAC). Although the experiments show an advantage of the proposed method in hard multi-agent tasks, it is critical that the authors better clarify the algorithmic contributions of the proposed method. Especially, it is important to distinguish the risk level controls of DRIMA from the algorithms it was built upon. Hence, my score for the paper in its current form is a “weak reject”. I am willing to raise it if the authors address the concerns pointed in my review.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces implicit quantile networks (IQN) into value-decomposition multi-agent reinforcement learning methods (QPLEX or weighted QMIX). The proposed method uses IQN for two components of QPLEX, for the joint action-value function and the transformed action-value function, respectively. The author expects IQN for the joint value function to account for environment-wise risk while the IQN for the transformed value function to account for agent-wise risk. 
main_review: ***Soundness***

The main concern of the reviewer is about the soundness of the proposed method. To be specific, the reviewer is not convinced that this method can disentangle the environment- and agent-wise risk.

a) Why does $w_{env}$ reflect environment-wise risk? Conditioning the joint action-value function on $w_{env}$ is not enough. The joint value function learns the Q function for the whole team and involves both environmental randomness and other agents' uncertainty. The authors may would like to consider defining agent-wise and environment-wise uncertainty formally.

b) Similarly, why does $w_{agt}$ reflect agent-wise risk? (b-1) The difference between the transformed action-value function and joint action-value function lies in their representational capacity. (b-2) The loss function for the transformed action function is more like a soft version of the QPLEX objective. Therefore, the reviewer thinks that $w_{agt}$ cannot reflect agent-wise risk, from the perspective of loss function and NN structure.

c) Current experiments do not support these claims either. In Fig. 5, the authors show that changing $w_{agt}$ and $w_{env}$ influences performance on MMM2, but this map is not enough as far as the reviewer is concerned. According to Fig. 7,  these risk levels have limited influence on other maps. Moreover, it is _very_ important but is absent from the paper how these risk levels influence the behavior of agents. The reviewer would suggest using a grid-world example and ablating $w_{agt}$ and $w_{env}$ separately, and then observing agents' behavior.

***Evaluation***

a) The reviewer can not tell whether the results are cherry-picked. 8m_vs_9m is not in the latest version of SMAC. Please show the results (comparisons against baselines) on all 14 SMAC maps. 

***Clarity***

This paper is not well written, and there are various grammatical errors throughout the paper. Some notations are not clear. For example, L_{td-IQN} is used twice (page 4 and 5) in the paper but stands for different loss functions. 
summary_of_the_review: The reviewer finds the problem under research in this paper interesting. However, the reviewer is not convinced that current methods can solve the problems and disentangle the environment- and agent-wise risk. Current experiments do not resolve this concern.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper describes a few different loss functions and heuristics to use
distributional RL in Dec-POMDPs in order to obtain risk-sensitive policies. 
main_review: 
The paper describes a few different loss functions that try to combine
distributional RL with decentralised reinforcement learning in
POMDPs. However, it does not demonstrate theoretically that these loss
functions achieve the intended effect, i.e. that they are a natural
outcome of the problem of decentralised distributed RL.  As such, the
paper is purely experimental.

The discussion of QTRAN and weighted QMIX is a bit peculiar. The
authors regard them as risk-seeking, while this is not explicit in the
cited papers. They draw this conclusion based on how the loss function
is defined, with OW-QMIX being 'optimstic'. But this is, to me, only
an intuitive understanding of the algorithms and does not arise from
them actually performing a risk-sensitive maximisation.

It is not clear to me why explicitly separating the two risk sources
helps in this scenario, or how it is achieved.  The authors mention
the sign of the TD-error as a way to obtain environment risk, but they
do not show that this is a reasonable idea.

So, viewing everything as purely heuristic, let us look at the
experimental results. In terms of the 'battle won mean' metric,
performance is competitive with the state of the art. 

Although the paper initially makes much of the problem of
disentangling risk, this is only analysed in a short paragraph in the
experiments. 

Note:
You say that the assumption is $\max_u Q_{jt}(s, \tau, u) = [\max_{u_i} q_i(\tau_i, u_i)$.
I generally fail to see how this can be the case,since $q_i$ is marginalising over all possible states.
I note that in (2), there is no $s$ in $Q_{jt}$.

summary_of_the_review: The heuristics are poorly motivated theoretically, and the experiments do not clearly highlight how disentangling risk sources can be helpful to drive risk-sensitive behaviour.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work introduces DRIMA, a Multi-Agent Reinforcement Learning algorithm that attempts to learn risk-specific behaviors, where risk is separately considered by its source: from other agents or from the environment. The authors explain their method as a novel combination of prior approaches and validate it empirically in the Starcraft Multi-Agent Challenge testbed. In addition, they provide some demonstration that non-disentangled risk sensitivity does not yield the same benefits.
main_review: # Strengths
- The performance benefits of DRIMA are clear and consistent -- at least in the SMAC testbed (which represents a challenging set of tasks).
- The authors compare against a good variety of baselines.
- The experiments include additional characterisation of DRIMA, beyond simply showing that it is SOTA. In particular, they look at how different risk choices impact performance.
- One insight is that, in the hardest setting, risk-seeking preferences are useful, both in terms of agent-based and environment-based risk. This naturally raises the question of whether risk-seeking is simply the important piece, but the authors demonstrate that risk-seeking *without disentangling risk source* is not particularly helpful. This provides a nice validation of the claim that disentangling risk is key to DRIMA's success.

# Weaknesses
- The methodology sections of the paper are very hard to follow. It often feels as though one needs a strong background with Centralized Training Decentralized Execution paradigms. In any case, I found that Section 3 left me with very little concrete understanding of how DRIMA works. However, owing to the rest of the paper being sufficiently accessible, I was still able to appreciate some of the contributions.
- For emphasis, I will reiterate that clarity of writing is the biggest weakness of the paper. One example is that I cannot connect the risk levels $w_\texttt{agt}$ and $w_{\texttt{env}}$ parameters (which I understand intuitively) to the training, execution, or design of the networks.
- The terminology use is confusing. In particular "transformed" and "true" action-values are not clearly explained, despite the amount of text they are given. There is lots of room for improvement here. The paper does not necessarily need more detail, but I think it would strongly benefit from a careful re-organization of Section 3. A general audience will not be able to understand the methodology. I am a MARL practitioner and I can't understand the methodology as written.


# Comments / Questions
- What is the best way to think of the reward, i.e. $r(s, \mathbf{u})$, versus the agent-wise utility? Is the agent-wise utility simply the estimate of the joint Q value given the observations available to the agent?
- If there is some way to visualize/inspect the way that disentangled risk affects the decision process or representation of value (compared to non-disentangled variants), that would be very interesting to see. Perhaps it is possible to compare how, for some reference state, risk is estimated in the disentangled versus non-disentangled representations. I do not think the paper strictly *needs* this analysis, but something along these lines could provide meaningful insight into what kinds of benefits DRIMA learning can create.
summary_of_the_review: My starting recommendation is: weak accept.

I think the empirical results speak for themselves, and the concept of disentangled risk is well motivated. In addition, I think the ability to separately tailor risk-preference based on the source of risk is an interesting contribution. Unfortunately, much of the contribution is hard to appreciate because various technical descriptions are vague and overly reliant on jargon.

I am confident that this paper could become a clearer "accept" if the communication were improved. In addition, some extra analyses to more deeply explore the impact of disentangling risk sources would validate the technique. I am happy to increase my score, to the extent that these suggestions are followed.
review_title: null

Meta-review: This paper proposes a method of multi-agent reinforcement learning that separately deals with the risk associated with uncertainties of the other agents and the risk associated with the uncertainties of the environment. This allows for example to be agent-wise risk seeking and environment-wise risk averse.  The proposed approach is largely heuristic with little theoretical justifications.  The experimental results are promising but not sufficiently convincing, given the lack of formalism.  Further improvement on clarity might complement the lack of formalism or theoretical justifications.
Decision: Reject