Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This article solves the Dec-POMDP problem, based on MAAC and QMIX algorithms. When the agents are homogeneous and optimized only through team rewards, it is easy to learn similar policies for each agent. This makes the multi-agent algorithm finally converge to a locally optimal joint policy. If each agent can complete different parts of the overall goal, the joint policy that converges to is obviously better. Many current works model the above problems as task assignment or role assignment problems. However, the agents with different roles in these works are basically rule-based, and the tasks are all manually defined. There are also some works that unsupervisedly generate different strategies and roles by introducing diversity constraints, but the generation process has nothing t o do with the task. In order to solve the shortcomings of the above methods, this paper proposes a MARL method based on reward shaping to encourage the division of labor between agents, and at the same time introduces two regularize terms to solve the problem of too similar agent policies at the initial training stage. At the same time, reward shaping and reinforcement learning are optimized simultaneously, forming a bi-level optimization problem. The paper also designed three tasks that emphasize division of labor to verify the effectiveness of the algorithm.
1. Neither the MAAC nor the QMIX algorithm on which the paper is based has good scalability. Although the independent learning algorithm is simple, it can achieve better performance on many tasks and has good scalability. This paper should additionally use independent learning algorithms as baselines, and apply the intrinsic rewards proposed in this paper to independent learning algorithms.
2. The three tasks used to verify the algorithm in this paper are all specially designed, with a strong emphasis on division of labor. I think the paper should additionally explain the limitations of the algorithm, such as which scenarios will be more effective and which scenarios will limit the learning ability of the agent.
3. The optimization process of bi-level problems is very unstable. The algorithm proposed in this paper contains many hyperparameters, and the sensitivity of the algorithm to hyperparameters should be shown in the experimental part.
review_title: Solve Dec-POMDP problem based on MAAC and QMIX. Interesting and timely work~

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper discusses an analysis of the emergence of individuality in a multi-agent system based on reinforcement learning. The emergence of individuality is based on intrinsic reward. The intrinsic reward is assigned to each agent by forcing them to be "different" according to a given distribution. The reviewer struggles to see this mechanism particularly insightful. This does not appear to the reviewer as emergence of individuality; his looks more like the emergence of different behavior (which can be seen as individuality, but it is more like forcing different behavior, not individual behavior, as interpreted normally when we study societies in my opinion).

Strength:

- According to the reviewer, any study that sheds a light on human phenomena using reinforcement learning is fascinating. 

Weakness: 

- The model used for studying the emergence of individuality is not convincing in my opinion. In fact the authors essentially impose the emergence of individuality from outside by giving a reward for being different. 

- The reviewer wonders if the emergence of individuality at the end of the day might actually emerge just by considering individual learning models. I would say that this is what you observe in many multi-agent systems with individual learning models. What is the main different in this case? This is not completely clear to the reviewer.

- The choice of the mechanisms (MAAC, QMIX, etc.) is not sufficiently discussed by the authors.

- The goal of the evaluation is unclear: the goal of this evaluation is not the emergence of individuality in my opinion.

In general, I would say that the question that the authors try to address is indeed a fascinating one. However, the reviewer really struggles to understand the choice of the mechanism selected by the authors. The reference to biology are not convinced. The authors say: "Biologically, the emergence of individuality is attributed to innate characteristics and experiences. However, as in practice RL agents are mostly homogeneous, we mainly focus on enabling agents to develop individuality through interactions with the environment during policy learning". Is it not something that happens if you have individual models in any case? It seems to me that the individuality/differences are artificially imposed on the system itself.

Overall, the contribution of the paper is not significant in my opinion, since unfortunately the work does not really provide insights about emergence of individuality, which is the main stated goal of the paper.  

Questions:

1. How do you map the mechanism used for the emergence of individuality to real situations? This appears in a sense as your goal, but it is difficult to see how it is possible to interpret the results of the simulation considering it as a simulation of a real-world society.

2. In the Introduction the authors say "Analogically, the emergence of individuality should also be essential for multi-agent cooperation". I would claim that this is difficult to prove by analogy as stated by the authors.

3. The goal of the evaluation is not completely clear. In fact you simulate a variety of games, etc. However, it is unclear how this  is related to the fundamental problem of the paper, which is the emergence of individuality.

4. Why do you use "positive distance"? What is the motivation of this choice?

5. The discussion about QMIX is not completely clear. Why is it necessary? How do you link this with the goal of the paper? Is there any clear mapping with real-world situations?


review_title: The topic of this paper is fascinating, but, in my opinion, the authors do not really provide an insightful model about the emergence of individuality

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper tackles the problem of exploration in multi-agent RL,
formulated as a Dec-POMDP. The authors propose to shape the reward
using the output of a classifier that tries to determine which agent
saw a particular observation. Moreover, the authors propose some
regularization schemes to "break ties" early on in the training. Then,
it is shown that the proposed reward shaping term can be integrated
into two popular MARL algorithms that use centralized training: MAAC
and QMIX. The integration into MAAC is relatively straightforward
because MAAC uses independent critics, whereas the integration into
QMIX is more involved due to the Q-function mixing step.

I believe this is a very important problem being tackled by the
authors. Even in single-agent RL, exploration is a major issue; this
only gets even more apparent in the multi-agent setting. I believe
that the paper was well-written, and the Introduction made it very
clear what exactly the paper's contributions were, which I greatly
appreciate.

Unfortunately, I'm not sure that I believe that the proposed bias is
truly useful. I understand the intuition: we want agents to
"specialize" their observations, such that it is easy to predict which
agent is receiving any particular observation. However, won't this be
counterproductive in most practically interesting domains? For
example, consider a team of robots working together in a factory or a
household; they will constantly be changing their environment as they
take actions toward operating the factory equipment or cleaning the
kitchen, which means the observations they receive will always be
changing. But the proposed reward shaping mechanism in this paper
would "fight" against this progress, because it would encourage the
agents to engage in trivial behavior just to be able to see the same
observations over and over. In my mind, it would be important to
consider the dynamics of how the classifier p(i | o_i) is changing
over time.

Another option could be to simply encourage the agents to learn
different policies, maybe measured via KL-divergence of P_{agent 1}(a
| s) and P_{agent 2}(a | s). I believe that this bias would be
sufficient to solve the example presented in Figure 1? Does this bias
seem reasonable, and if so, how does it compare against EOI?
Basically, more broadly, I would have liked to see experimental
comparisons that convince me that EOI is the *correct* bias to use
versus other natural biases, whereas the current experiments only seem
to compare against non-shaping methods like ROMA and HL.

Some other questions:

1. How significant is the fact that you are ignoring second-order
effects in solving the bi-level optimization in Appendix A.2? Have the
authors conducted preliminary experiments to prove that it doesn't
make much difference? I understand that this practice is standard in
methods like MAML, but it would be nice to verify that it doesn't
matter in this setting either.

2. Looking at the shaped reward computation, r + alpha * p(i | o_i),
it seems like if the classifier were naive and simply outputted a
uniform distribution, you would still be giving positive intrinsic
reward in that case. Might it make more sense instead to consider the
divergence between p(i | o_i) and the uniform distribution?
review_title: Important problem, but unsure if this bias makes sense

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper contributes a method based on reward-shaping to encourage the emergence of distinct agent behaviors in fully-cooperative multi-agent reinforcement learning (MARL), within the paradigm of centralized training with decentralized execution. They propose to learn a classifier that predicts agent identity given the agent's observation, and use the classifier probability as an intrinsic reward that is added to the environment reward. As such, there is a positive feedback loop whereby agents are rewarded for being distinguishable and hence generate data that is easier for the classifier to predict correctly. They provide two regularizers to help kick-start the positive feedback loop when agents' policies are similar at initialization. The paper implements the method, called EOI, on top of two base MARL algorithms. Using three illustrative environments where optimum team performance requires distinct behaviors among agents, the paper shows that 1) EOI outperforms the two base algorithms and other baselines that address exploration and diversity; 2) the two regularizers improve performance and help the intrinsic reward to serve as an inductive bias; 3) agents do perform distinct behaviors. They also show comparable performance on a benchmark StarCraft II micromanagement task.

I give a rating of 5 for the following reasons. Classification-based intrinsic reward have been used for diversification in MARL in previous work (Lee et al., 2020, Yang et al. 2020), but this paper provides a different formulation along with effective heuristics to speed up the initial training of the classifier. The experiments are clear in showing the benefit of EOI and EOI + regularizers when built on top of the two base methods.
However, there are main points of concern. I am willing to raise my score if the main points below are sufficiently addressed.

This paper incorrectly positions itself against the previous works Lee et al. (2020) and Yang et al. (2020). In section 2, the authors say "none of these studies appropriately link behavior diversification to the success of the cooperative task". However, in section 3, the authors say that "EOI directly correlates individuality with the task by intrinsic reward", making it seem that EOI differs from Lee and Yang in that regard. This is not true. The intrinsic reward in EOI is the probability of correct classification of an agent identity. The intrinsic reward in Lee and Yang are based on correct classification of a latent skill variable. Neither agent identity nor a latent skill variable are directly linked to task success. The intrinsic rewards in EOI and in these previous work encourage the emergence of distinguishable behavior; none of them have any direct link to task success. The authors either need to explain their claim in more detail, or amend the paper.

Another main concern is about the case where EOI is built on top of QMIX. The authors define $Q^p_i( o_i, Q^a_i(o_i))$, which takes in the vector of individual action-values $Q^a_i(o_i)$, and define a TD error loss with the intrinsic reward $p(i|o)$. In the first equation in Section 3.3, the TD error serves purely to _estimate_ the expected cumulative intrinsic reward under the current policy. It does not find some Q-function that _optimizes_ the expected cumulative intrinsic reward. However, the initial formulation on page 3 (where the total reward is the environment reward plus the intrinsic reward) implies that the agents should maximize the expected cumulative intrinsic reward as well. Therefore, it is not clear at all how the intrinsic reward is being maximized in the QMIX case.

Points that did not affect the decision, but where the paper can be improved:
1) All equations should be numbered for easier reference.
2) First sentence in abstract needs a language check.
3) The introduction says "...agents tend to visit their own familiar observations". It seems that this has a negative impact on the exploration by an arbitrary agent. Authors should include more discussion.
4) Section 3.3, a more succinct way is to say that assigning the intrinsic reward to the specific agent bypasses the multi-agent credit assignment problem.
5) In Section 3.3, it is not clear what the authors mean by saying that the individual value (in QMIX) has no "actual meaning". Please be more precise.
6) At the end of Section 3.3, the authors say that due to off-policy learning, $p_{\phi}(i|o)$ is recomputed for each sampled batch. Is it recomputed purely because the current classifier is presumably more accurate? Is there any potential problem with doing so, in the context of off-policy learning?
7) All plots need thicker line width, including the legend. Colors need to be more distinguishable from one another. Currently, it is hard to see which curves correspond to which algorithms.

---
Edit: score increased from 5 to 6
review_title: Method is shown to be effective in experiments, but there are concerns about positioning relative to existing work and correctness of implementation

Meta-review: This paper introduces a method to increase diversity/individuality of agents in a MARL setup, based on intrinsic rewards coming from a classifier over behaviours.

Reviewers tend to agree that this is an important/interesting problem, which is related to exploration, a central problem in reinforcement learning. Several reviewers point out that the paper is well written. I appreciate that the authors have been responsive to reviews and have answered and/or addressed several points of concern of the reviewers. The proposed method performs well on the experiments carried out.

Reviews still point out several things that could be improved. The experiments mostly report reward curved, and only few results are actually clearly pointing out the individuality between agents. The fact that this method outperforms the baselines is good, but does not prove individuality and may simply be due to the authors spending more time on the tasks, or other undiscovered phenomenon.
A reviewer is concerned that this extra reward could encourage trivial behaviours, and it seems clear that it will if the relative weight of the intrinsic reward is too high. This should be discussed more.
Finally, a reviewer points out that classifier-based intrinsic reward for diversity already exists in published works and that this paper is incremental work.

The average score for this paper is very close to the acceptance threshold, but based on the reviews I recommend to reject this paper for ICLR 2021. I am confident that when the authors address further the reviewers concerns and improve the experimental results, this paper will be published in a future venue.
Decision: Reject