Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the problem of computing "diverse" joint policies in Decentralized POMDPs (Dec-POMDPs) with public actions. The paper proposes a method based on off-belief learning augmented with "repulsive" fictitious transitions to encourage diversity. Finnally, the proposed algorithm (ADVERSITY) is run on the card game Hanabi to produce new playing agents with diverse play styles. 
strength_and_weaknesses: STRENGTHS

- The problem studied in the paper is interesting for the community focused on the designed of learning agents in games, as well as for the multi-agent reinforcement learning community.

- The paper is well written and all the results are adequately commented.

WEAKNESSES

- While the paper has a mainly practical focus, I would have liked some theoretical analysis and/or formal claims explaining how the method is supposed to work. The experimental results seem promising in Hanabi, but further evidence is needed to conclude that the method could be applied to other domains.  
clarity,_quality,_novelty_and_reproducibility: CLARITY: The paper is well written and easy to follow.

QUALITY: The paper is lacking some theoretical evidence as to why the proposed method is supposed to work well in general settings.

NOVELTY: As far as I am concerned, the proposed techniques are novel.

REPRODUCIBILITY: The results presented in the paper are reproducible and the authors plan to make their code accessible open source.
summary_of_the_review: Overall, I think this is a good paper on an interesting topic. I only have some concerns on the theoretical groundings of the proposed method.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: A natural way to train qualitatively different policies for cooperative tasks is to penalize a new joint policy for performing well when individual components of this policy are paired with existing policies.  While this approach can identify multiple "incompatible" solutions to a cooperative task, this incompatibility may be achieved by learning to identify the behavior of existing policies, and then intentionally "sabotage" cooperation.  Experiments conducted in this work demonstrate just this type of behavior in the Hanabi environment.  To address this issue, this work presents the ADVERSITY algorithm, which modifies the existing "off-belief learning" algorithm to train joint policies that are incompatible with an existing policy, but cannot reliably identify that policy in order to sabotage play.

The ADVERSITY algorithm proceeds by rolling out the joint policy being learned.  At each step within this "real" trajectory, rather than computing the reward based on the true trajectory, ADVERSITY (like OBL) samples a "fictitious" trajectory that is consistent with an individual agent's observation history, but is generated by the existing policy that the new policy should fail to cooperate with.  In this way, during training, the new policy for an individual agent can never be sure whether it is interacting with the new joint policy, or whether it is interacting with the existing policy.  It therefore has no incentive to intentionally sabotage cooperation.

Experimental results demonstrate that compared to directly penalizing self-play agents for performing well when paired with existing policies, policies learned with ADVERSITY are far less likely to engage in such sabotage, and are more effective in cross play with other ADVERSITY policies trained with different random seeds.
strength_and_weaknesses: The key contribution of this work is the exposition of the issue of partner sabotage when attempting to train mutually incompatible policies as a starting point for ad hoc cooperation.  They also contribute the first method that attempts to address this issue, by leveraging off-belief learning.

The main weakness of this work is the relatively limited experimental evaluation of the proposed method.  While the experiments clearly demonstrate that the ADVERSITY algorithm is able to avoid learning sabotage behavior, they do not evaluate the algorithm as part of a larger stack for learning policies capable of playing Hanabi with a variety of partners.  A more thorough evaluation, where ADVERSITY is used to train a population of different agents, which are then used to train ad hoc policies, might indicate whether the diversity encouraged by the ADVERSITY algorithm is actually helpful for ad hoc play.
clarity,_quality,_novelty_and_reproducibility: The paper is technically sound, though the underlying methods are not particularly novel, as they are a relatively simple modification to the existing OBL algorithm.  What is more novel is the identification of the sabotage issue itself.  There are a few areas where greater detail would be helpful, both for clarity and for reproducibility:

- The process of sampling fictitious trajectories from the belief distribution is not described in sufficient detail.  While this method comes from previous work, it is critical to the algorithm, and so a brief discussion of the details of this process, particularly answering: 1) what is the loss used to train the belief model and 2) does this model require the ability to simulate transitions from arbitrary states?

- More detail on the underlying RL algorithm could be provided. For example, presumably the fictitious transitions are added to a replay buffer, but is there any prioritization done over this buffer?

- Algorithm 1 is a little unclear.  It isn't obvious that the loop is being run over the length of a single trajectory.  The update on the second-to-last line also suggests a tabular Q-update, rather than a gradient descent step.  It might help to split Algorithm 1 into two algorithms, one for the outer RL loop, and another for the fictitious sampling process.
summary_of_the_review: The main justification for acceptance is the fact that this work is the first to address the problem of "sabotage" behavior when training diverse sets of policies for cooperative tasks.  They clearly demonstrate that their method is able to largely solve the sabotage problem in the Hanabi environment.  That said, the empirical results are limited, and are not sufficient to demonstrate that ADVERSITY can be used to learn "good" strategies for Hanabi in ad hoc play.  Additionally, the ADVERSITY method itself is a relatively straightforward combination of two existing approaches.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on producing strong and diverse policies in Dec-POMDPs. One important aspect of the proposed method is to prevent the adversary agent from identifying whether it is self-playing or cross-playing by randomizing between the two scenarios. It is also important to bear in mind that in the studied setting in turned based with public actions. The proposed method will lead to having several highly-skilled agents with different styles of play.
strength_and_weaknesses: Tackling the problem of MARL with diverse agents has given little attention in MARL. Authors try to fill this gap in MARL by introducing ADVERSITY which relies on off-belief learning to put more weights on actions with meaningful information about the trajectory.

Questions:  
1- How was the SPWR baseline optimized in terms of hyperparameters or lambda? Using the same hyperparameters as ADVERSITY won’t be necessarily the optimal ones for SPWR. And for lambda, why 0.25?

2- According to table 1, the difference between SPWR and ADVERSITY in SP is not statistically significant. Could you discuss this?

3- Results are averaged over 3 seeds which is low. Increasing the number of seeds could be helpful in terms of trusting the results and see how the proposed method is robust against various scenarios.

4- What does “reasonable” mean in the abstract?
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly, and is easy to follow. Authors did a good job in explaining their motives, method, and results. Regarding the reproducibility, in the abstract it is mentioned that the source code will be released publicly, but it was not shared in the supp material.
summary_of_the_review: See above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an approach to compute strong and diverse joint policies in a Decentralized MDP. The proposed algorithm is then tested in the Hanabi card game to produce agents with diverse policies.
The problem faced in the paper is relevant and the paper is well-written, easy to follow, and technically sound.
The reviewers raised some issues about the lack of theoretical analysis, a limited empirical analysis, and a limited novelty.
However, after reading each others' reviews and the authors' feedback, the reviewers solved most of their concerns and agreed that this paper deserves publication.
The authors need to implement the suggestions provided by the reviewers while preparing the camera ready.
justification_for_why_not_higher_score: The lack of theoretical analysis and an empirical analysis limited to one domain make this paper not deserve an oral presentation.
justification_for_why_not_lower_score: The paper contains significant results that can be of interest to the ICLR audience.
However, I am fine also with a poster presentation.
Decision: Accept: notable-top-25%