Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper study the relation between role diversity of tasks and the MARL model performance. The role diversity (difference among agents) is described from three perspectives: policy-based, trajectory-based, and contribution-based. The authors analyze how the role diversity impacts the MARL performance in theory and experiments. The empirical results show a strong relation among the three metrics and the three main topics in MARL (parameter sharing, communication, and credit assignment).
main_review: Strength:
+ The paper provides an interesting perspective (role-diversity) to measure the difference between MARL tasks. 
+ They analyze the impact of the role diversity on MARL methods both theoretical and experimental.  The experimental results, measured by the three role diversity metrics, answered a common concern question that why the MARL model performance varies across different tasks.  
+   The discovered guidelines about the choice of training strategies will benefit the community.

Weakness:
- The generalization of the proposed metrics. The trajectory-based role relies on the observation overlap percentage. The observation overlap percentage can be easily defined and measured in SMAC, but how to measure it in other realistic scenarios, e.g., embodied vision robot? The contribution-based role highly depends on the output of the RL model. However, there are a great number of hyper-parameters that will impact the output, and different methods may be of different hyper-parameters. So, how to standardize these metrics for a fair comparison? 
- The experimental results are unclear. For example, in Table 1, only the policy-based role diversity is reported in MPE. Why not report the three metrics jointly for a more comprehensive analysis?  similar problems do also exist in Table. 2 and .3. These can help us better understand the difference between the three metrics.
summary_of_the_review: The paper is well-motivated and interesting. The provided guidelines are practical. But I am concerned about the generalization of the proposed metrics in other MARL environments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work looks at defining role diversity as a means for analyzing multi agent dynamics. It proposes three different perspectives on analyzing roles in multiagent systems. One is from a policy perspective (based on KL divergences), an trajectory perspective (how much do the agents overlap in what they observe), and a team-contribution perspective (weighing the effects of the agent to the team reward). 
main_review: 
Strengths: 

* Interesting question of how to define roles in multiagent systems and apply them to analyzing policy behavior. 

Weaknesses:

* The paper is not well written. The paper does not do a good job at motivating why we should care about analyzing multiagent systems from the perspective of roles, or the alternative role definitions they propose. 

I would recommend the authors re-frame the paper narrative and improve the writing quality; I’d made suggestions under “Summary of review”. 

* The paper does not do a convincing job at showing that the roles are core to “cooperative training strategies for multiagent RL”. 

Assuming we do think roles are an important way of understanding multiagent behavior, the role perspective only shines light on agent patterns, but don’t necessarily cause improved cooperative strategies. A convincing suite of experiments that would need to be conducted would be if agents were explicitly optimized for those particular metrics with minimal impact to the nominal performance. That would be the right test to show that the decomposition does in fact play an important role in policy optimization, which is what the authors claim in their abstract. 

* The theoretical analysis is not well-motivated and it’s not clear whether it provides any operationalizable insights. It adds more confusion to their role diversity framework. 

For example, the authors try to show that their policy-based and contribution-based role can be tied to an approximation error between the optimal joint and independent policies. What’s confusing is that their analysis suggests that agents should minimize these role metrics in order to perform closer to the optimal joint distribution. This conclusion doesn’t make sense since the work’s experimental setup and narrative is that these metrics are important in improving the policy quality.


summary_of_the_review: 
--- motivation --- 
I’m not convinced that better understanding roles will lead to improvements in multiagent coordination. In fact, this is not really the investigation the paper conducts: The paper demonstrates there are certain correlations between their role definitions and coordination performance, but not that this understanding results in coordination improvements. What concrete challenges currently exist in multiagent optimization --- e.g. what do the authors mean by `roles can avoid the issue of “policy degradation”` in the abstract? --- and how can these challenges be addressed by understanding how groups/teams can be factorized into roles? 

I’m confused why the authors break down MARL cooperative algorithms by “parameter sharing”, “communication” and “credit assignment”. These are components that don’t seem directly comparable to each other, and it’s not clear how roles are related to the components. Parameter sharing is a technical means to learning policies, communication is an assumption by which we allow additional agent interaction / information passing, and credit assignment is a general challenge when optimizing in long-horizon settings for reinforcement learning. 

The authors directly propose their three different role definitions in the second to last paragraph of the introduction, but these alternative definitions need more context and motivation. It’s not clear what challenges they address and why these definitions are ones we should use for characterizing multiagent behavior. Additionally, without context or grounding in prior work, it’s unclear what “policy-based,...” roles should even refer to. 


--- method --- 

How does this proposal show that agents take on the same role? If I’m finding a good position just like all other agents, shouldn’t this indicate that we have the same role? This doesn’t seem reflected in this metric. The policy-based role also seems to have a lot of overlap with the trajectory-based role --- in some sense, the trajectory based role feels quite redundant: If I can infer what role the agent has, I should be able to have a good generative model on what the agents’ most likely trajectories should look like. 

In Equation 1, a sum over the policy conditioned on an action is an unnormalized probability distribution. It’s not clear to me how this is being used in the KL metric. 

In Equation 2, the normalization should happen over A^2, and not A. 

It seems weird that the trajectory-based role is defined as the observation overlap. Observation overlap seems to be a measurement of … pairwise observability (i.e. how much information do I have about what you see, and you about what I see?), and less about roles. 

I disagree with the authors on their interpretation of Q values on “contribution-based roles”: in the setting where agents are performing individual policy optimization, I don’t see how Q values [optimized to reflect the agents’ expected return, and not a ratio between the outcomes of an agent’s actions and all other teammates] should not be interpreted as the agent’s contribution to the team. If there’s a misunderstanding or some related work that asserts this, the authors should elaborate. 

--- theoretical analysis --- 

The notation is not self-contained in this section, e.g. “Please refer to Sec. B.1 for the detailed definitions”. 

The second term in the error is the irreducible error after we find a best fit for relating the optimal joint Q values and the concatenation of independent Q values. How do we go from this line to line (5)? On a related note, how are we going from line (5) to (6)? 

“The first term on the RHS of (5)...” I don’t see how this difference is related to the contribution based role which is based on an agent’s Q values. How can this be related to the linear weights? 

“The first term on the RHS of (6)...” Minimizing the bias results in minimizing the policy-based metric, which intuitively leads to agents _not_ aligning their actions. Does it make sense to think about this minimization as reducing the error we incur when approximating a joint policy with independent policies? This seems contradictory to what the authors want to propose in their methods section. The bias term should also normalized over the number of samples too. 


--- claims --- 
“These definitions are intuitive and cannot accurately describe the role difference.” Why not? What is insufficient with their definitions? It would be nice if these definitions could also be used in the experiments to compare and contrast the success/failure modes.

“As common sense would indicate, actions taken at the same time step can indicate different roles” - Do actions do this? This is arguable. Actions can be considered coordinated or uncoordinated, joint vs. independent. They can indicate different or shared roles.

--- figure ---
Figure 1 is quite confusing for the following reasons 
> What is the blue vs. green color schema supposed to refer to? 
> I don’t understand what MARL alg 1 and 2 should represent. I thought the agents are trained using the same algorithm? 
> Related to my comment on “why the authors break down MARL cooperative algorithms by ... ”, I don’t understand why these axes are being used as input parameters for the algorithm in Fig 1a, and then as outputs from Role Diversity.
If the point is that role diversity should inform our assumptions in developing algorithms, then I’d recommend rephrasing the figure caption, omitting the performance graphs, and improving the figure pipeline. The figure doesn’t elucidate the authors claim that “role diversity [avoids] possible bottleneck of a MARL algorithm … ” What is this bottleneck? 

Figure 2 onwards refer to the scenarios with hard-to-parse names, eg. “1s1m1h1M_vs_5z”. These figure names should be updated for clarity.



--- writing nits ---
Introduction
> “lack the study of the question of why” - too wordy and awk. 
> “In other word” → “In ”
> “Current researches are more focusing”
> “The reason maybe:” - informal. 

Related works
> “adopt q-learning” - note capitalization
> “In this study, we contended that” - note tense
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: To address the problem of algorithm choosing in different MARL tasks, this paper proposes to use role diversity as a metric to describe MARL tasks. For choosing parameter sharing, communication mechanism, and credit assignment strategy, this paper defines three role diversity metrics, i.e. policy-based, trajectory-based, and contribution-based, respectively. They also find that the error bound in MARL can be decomposed into three parts that have a strong relation to the role diversity. To evaluate the proposed method, they further conduct some experiments on MPE and SMAC environments. 
main_review: The reviewer finds the paper is rather timely and interesting. With so many algorithms being proposed, how to choose one is quite challenging sometimes. This paper proposes to use role diversity to describe the MARL tasks and the experiments strongly show the relationship between algorithm choosing and role diversity. However, the reviewer also thinks the paper needs to be improved.

(1) The paper does not provide enough discussion or experiment about the connections and differences between these three role diversity definitions. They might describe different diversity aspects. But there also might be an underlying diversity definition behind these three definitions. Another question is that is one role diversity definition (such as policy-based) for one algorithm design (such as parameter sharing)? Are there any connections?

(2) Theoretical analysis is not clear enough. The reviewer appreciates that this paper provides formal analysis, but the reviewer is confused about how the empirical role diversity definitions are linked to theoretical analysis. Specifically, more explanations about Eq. (5-6) are expected.

(3) The definitions of trajectory-based and contribution-based diversities should be explained in Eq. (3-4). For example, what are the definitions of $d_T$ in these two equations? How to compute them?

(4) The discussion of how to obtain an accurate measurement of the role diversity before training in Sec. 6 could be moved to Sec. 3. 

(5)  What do the figures in Fig.2 (b)(c) want to say? 

(6) Some minor errors. It should be $T$ instead of $t_0$ when describing the notations of Eq. (2).
summary_of_the_review: This paper is timely and interesting, but there are still some concerns, especially the the first one in the Main Review. The reviewer thinks this paper should be further improved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a role diversity metric to quantify the difference between agents in multi-agent reinforcement learning. The metric is used to inform the use of communication, parameter sharing, and other MARL algorithmic decisions.
main_review: 
-- Positives --
- Role diversity is important in multi-agent environments. Being able to quantify it might allow us to better understand how to improve multi-agent reinforcement learning. I also agree that such a metric can help us understand why some MARL algorithms perform so differently across a range of tasks.

-- Negatives --

1. - (3.1 - Policy-based role): The frequency of actions does not seem to be a reliable metric for determining the role of the agent for the following reasons: i) there is no guarantee that actions do the same thing (e.g. action '3' can mean different things for a "marine" and a "zergling" in the game of SMAC), ii) not considering the _order_ that the actions are selected loses important information (e.g. in a maze, going left and then right is significantly different to going right and then left). Therefore I believe focusing on the trajectory as a whole makes much more sense and includes the frequency of the actions of this paragraph.

2. - (3.2 - trajectory-based role): The "overlap percentage" of the observations is not general enough and does not seem straightforward to calculate in all environments. The authors present an example for the SMAC task in the appendix, but SMAC happens to have a circular observation radius which is then easy to calculate. There are many environments where the observations cannot be easily compared (e.g. a simple image, or even features that describe the state of the environment without a specific physical representation). In addition, fully-observable environments are not uncommon and, in my understanding, the rule breaks down in that case (the distance between observations is always 0).

3. - In general, all those metrics seems to require that training has already been completed. The contribution-based role specifically requires (trained, because I assume untrained aren't useful) q-values. As such, I cannot see how these metrics can help with the training of MARL agents, since they require those trained policies, to begin with. The same should apply to the policy-based rule. In my understanding, this is not in line with the motivations and promises of this work, that a diversity metric can help with a better training strategy and performance in MARL. 

Other/fixable:

4. - (5.1 parameter sharing): I would expect different roles to require different parameters and roles that are similar to benefit from shared parameters (which is also shown in "...selective parameter sharing..."[7]). You could consider a similar parameter sharing scheme on top of the "no/partly/fully" shared parameters.

- While the writing is clear, and conveys the author's intentions, it could certainly be further polished. There are some sentences that are vague or that could benefit from rewriting (e.g. first page, "In other word, even adopting the ... performance", but there are other sentences as well). The document could benefit from some proofreading.

- The bibliography is untidy and needs to be further polished. Many of the references inaccurately state "arxiv" while they have been published in conferences. 
- Using citations as a noun is discouraged (e.g. "In [52], the role]...")
- (page 5) acroess -> across
summary_of_the_review: For the stated reasons, the proposed diversity metric does not manage to convince me of its potential usefulness, although I still believe that the research direction is promising.
review_title: null

Meta-review: This paper proposes a Role Diversity metric, meant to quantify how different roles are in a multi-agent RL setting. There's actually three versions of this metric, or three aspects (the distinction is not entirely clear to this area chair).

The reviewers are generally not very enthusiastic about the paper, with scores hovering at or just below the acceptance threshold. There has been extensive discussion between reviewers and authors, but there a sense that there is confusion about the exact purpose and contribution of the paper. This is reinforced by the authors' "letter to area chair", which outlines several ways the reviewers have not gotten the message. Reading the paper, it appears to me that the root cause is that the authors are indeed not communicating clearly what the paper contributes and why. It is, after all, the authors' responsibility that the reviewers understand the work. My own impression is that the text is dense and not particularly easy to get through. Perhaps the authors are simply trying to cram too many contributions into a single conference paper? This is a classic error which leads to hard-to-read papers. In addition to this, there is a lingering concern about the generalizability of the proposed methods.

I think the authors need to work more on their presentation, and perhaps reconsider which parts to include in their paper and exactly which measure they want to send, before they submit to another venue.
Decision: Reject