Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes new algorithms (DiL-piKL, RL DiL-piKL, Diplodocus) for decision-making in games requiring cooperation with humans. The proposed method combines a number of ideas (self-play, leveraging imitation learning with KL-divergence regularization) to produce an agent that learns to play well against (and in-cooperation with) human players in a tournament of no-press diplomacy. Theoretical justification (bounded regret under certain assumptions) is included. The experimental results show the agent performs well compared to human players, measured by final rank as well as subsequent analysis of gameplay.
strength_and_weaknesses: Strengths
  * The paper tackles a problem that is less well-studied (compared to typical single-agent environments.)
  * The paper builds on previous works methodically. The proposed algorithm is intuitively clear. Each idea is well-motivated and the overall algorithm is easy to follow.
  * The experiments are well-constructed and nicely detailed. A number of strong baselines are used.
  * The overall algorithm, Diplodocus, clearly improves the state-of-the-art. 

Weaknesses
  - I didn't spot any major technical issues with the paper. A couple of minor suggestions and questions.
  - How might a standard MCTS (AlphaGo style) fare? While unsound for the reasons mentioned, it might make the experimental section more complete.
  - What would be needed to make the error bars in Table 2 smaller?
  - I'd be curious to better understand how performance varies as a function of human player level. For example, can Diplodocus outperform weak human players? This would likely require a more complex tournament setup and is likely outside the scope of the paper.

clarity,_quality,_novelty_and_reproducibility: Clarity
  * The paper is well written. The ideas are well structured and nicely detailed.

Quality
  * The paper is technically sound. It builds on prior work. The proposed ideas are shown to be effective in experiments. I think this paper would be a nice addition to the literature on agent design in games requiring cooperation with humans.

Novelty
  * The paper contains a number of technical contributions. Diplodocus is a novel algorithm, as far as I can tell.

Reproducibility
  * There seems to be sufficient detail included. Including code would have been better.
summary_of_the_review: The paper contains a new algorithm for training agents that can play well in games requiring cooperation with other human players. It seems to extend the state-of-the-art in this area although the error bars are a bit too large. The paper is well written and is technically sound.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper considers multi-player games, in particular No-press diplomacy. A new algorithm is proposed that combines self-play reinforcement learning with a no-regret learning algorithm that is regularized by human play.

Theoretical properties of the no-regret algorithm are proven bounding the cumulative regret for the multi-player case, and showing convergence to Bayes-Nash equilibrium in the two-player case.

Experiments with human players show that variants of the algorithm perform better than some baseline algorithms, and even on par with the expert human players included in the tournament. The human analysis of the computer players' behavior reveals that the proposed variants elicit more human-like cooperation strategies. 

strength_and_weaknesses: The proposed algorithm is sound. It has some novelty, although it is mostly a combination of existing algorithms. There is also a theoretical analysis of the DiL-PiKL component.

The empirical performance seems also impressive, and the expert analysis certainly sheds further insights into the strength and weaknesses of the algorithm variants. 

It is clear that regularization is mainly added to inject some cooperation flavor into the algorithm. A large chunk of the human play does not involve cooperating moves, but moves that aim to improve the position of the player (maximize the reward). Because regularization is uniform across all moves, it is forced to be small enough to avoid drop in performance (even if the human moves are reasonably good), which leads to weaker and less coherent cooperation, when needed (as indicated by the experts). 

clarity,_quality,_novelty_and_reproducibility: The paper is well written. The approach is mostly combination of existing techniques, but the strong performance of the proposed algorithm make the paper a significant contribution. The results are probably difficult to reproduce, which is not unexpected due to the complexity of the algorithm. 


summary_of_the_review: The paper proposes an algorithm with theoretical guarantees and strong empirical performance in a difficult and complex domain. While the components of the algorithm are not very novel, the paper still provides a significant contribution.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Left to their own devices agents will learn equilibrium specific to the coplayer(s) that are encountered throughout their training algorithm. This can be problematic in (semi-)cooperative games where successful equilibrium selection depends on a population of players distinct from those that were featured during training. This manuscript investigates this problem by attempting to learn "human-like" equilibrium selection in the many-player general-sum game of no-press Diplomacy. They extend a previous line of work that proposed an optimization objective of a weighted combination of being performant and human-like. They consider generalizing the fixed weighting into a sampling process so that experience is generated on a spectrum of performant and human-like. They test a suite of agents on a set of paid self-volunteered humans of indeterminant strength (likely quite good in certain variants of Diplomacy; note each variant is very different). Their proposed agent comparable similar to the best human in the population alongside a simple BR(BC) baseline. 


strength_and_weaknesses: **Strengths**
 - This paper is exceptionally well written. It provides an accurate and concise narrative of all modern work (ie., those containing DRL; it would be nice if they referenced some pre-DRL work in this space) in a very easy-to-process exposition; especially, as it pertains to the motivation behind each innovation. 
 - The innovation focuses on a real problem: fixed-coefficients in compound losses limiting the scope of discoverable strategies. The propose a reasonable solution of sampling throughout training and theoretically show that in certain cases this maintains convergence garuntees. The convergence garuntees are for their modified objective and not the true objective, and I would be interested to hear the author's thoughts on why this is reasonable and how their work might also show convergence to the true objective? The true objective should be to learn an equilibrium (maximize utility), and the human-like component of the loss should only seek to influence selection from equilibrium. 
 - The paper also contains empirical results on full no-press Diplomacy with agents and humans including qualitative results. They perform notably strong against within the agent population, but the quantitative results including humans are inconclusive. The qualitative results are mixed, but generally positive towards the proposed agents.  While the results are not overwhelmingly positive it's good to see such diverse and interesting analysis. I would suggest that the authors also consider in future work inquiring from human-players about finesse or attempts to identify the agent player during play. Perhaps have some probability that no agents are present?

**Weaknesses**
 - A large concern I've had with this work and its predecessors is that it concerns the equilibrium selection problem without adequate discussion of its tradeoffs. This work penalizes an agent towards behaviors similar to a single representative policy that are meant to represent an entire population of interest. Artifacts of construction of the representative also play a large role in the equilibrium selection process by are not discussed. Why should a single policy be sufficient to represent a target population of interest? What about biases about the representation of individuals within the population as it pertains to the representative? How would this effect performance across diverse sub-sets of the target population and to those outside of the population?
 - Could the authors please elaborate on why the beta distribution should be considered common knowledge? (Sec 3, Par 2).
 - This work proposes effectively using a distribution over behaviour policies to generate training data. I am curious how the authors arrived at the coefficients that represent the types within this population? Moreover, how diverse are these policies? I should expect that adding diverse experiences would cause the agent to require much longer training, but Fig3 makes it seem that the agents require effectively no training? 
 - As the authors are aware the country played has a large impact on the success of an agent (Appendix). This makes comparing agents on such a small number of games challenging when attempting to normalize out the power that they played. The Elo bias numbers associated with each power that the authors construct are so large that would render any attempt to understand Table 2 unfair. I would have preferred that more controlled studies were performed to really try and understand the differences with the least confounding variables. I do appreciate that human studies are very expensive and the results within are still nice.
 - The claim of "Mastery" is exceptionally strong considering the inconclusive evidence about this method being measurably different from vastly simpler baselines.
 - The main methodological change is related to influencing the human-likeness coefficient, but the experimental results do not focus on trying to understand the changes, but instead solely care about a particular game. 
 - It's not clear to mean that being human-predictable is a good objective to optimize towards. Consider what Expert 1 said (pg13): "I think it is quite predictable in what motivates ... That's potentially exploitable, even with the strong tactics". This is a very astute observation, and I tend to agree with the underlying idea. We would like to select human-like equilibria for cooperative, but this may hinder the policy's performance when cooperation is no longer needed. A particularly interesting problem in Diplomacy where the amount of cooperation required is fluid over the course of the game. 
clarity,_quality,_novelty_and_reproducibility: This paper is exceptionally clear and well written. It contains a novel result with theoretical and empirical (quantitative & qualitative) results. 

This work is likely impossible to reproduce without the code being released; but, moreover, it contains human evaluation. 
summary_of_the_review: This paper presents an improvement for the human-like equilibrium selection problem. It demonstrates its efficiency with a in-depth study of the game of no-press Diploamcy. The results are not convincing enough that mastery was achieved, but the investigation is sound and this is a noteworthy milestone towards solving Diplomacy. 

Therefore I think it is worthy of publication, but I strongly encourage the authors to tone down the claims of its accomplishments, because it undermines the work and credibility.
review_title: null

Decision: Accept: notable-top-5%