Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work reinterpret LOLA as approximating a proximal operator, and then derive a new algorithm called Proximal LOLA (POLA) which uses the proximal formulation directly. The updates of the POLA method are parameterization invariant, which means that behaviourally equivalent policies can result in behaviourally equivalent updates.
strengths_and_weaknesses: Strengths:

- This work provides good insights for improving the LOLA method in partially competitive environments.

Weakness:

- This paper is poorly written and hard to follow.
- The experiments in this paper are not solid enough. And in Section 4, the author only states the experement settings and the results, without analyzing and explaining the results of the experiment.
questions: - In the results of Full History IPD experiment (Figure 4), there is a significant degradation in the performance curve of the POLA-DiCE method. Is there any specific analysis and explanation about this phenomenon?
limitations: This work is an incremental work based on the LOLA method, simply combined with the Proximal Point Method for improvement, and does not have enough novelty.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes POLA, a MARL agent that can learn reciprocal policies in general sum games such as iterative prisoner's dilemma (IPD). The proposed algorithm is largely based on LOLA, a MARL agent that can learn reciprocity-based cooperation in the IPD game. The main contribution of this paper is that under certain parametrization of the policy space, LOLA may fail to learn good policies. The paper highlights that a more ideal learning algorithm needs to be invariant with the policy parameterization. Although it is impractical to implement such update rules, the paper borrows ideas from PPO and proposes a proximal update version of LOLA. The proposed algorithm demonstrates better performance on IPD and the coin game.
strengths_and_weaknesses: The main merit of this paper is to use a proximal update objective to improve LOLA. Although proximal policy update has been largely explored in the RL community (e.g., TRPO, PPO), it is not previously applied to LOLA. While the combination of LOLA and proximal method lead to better performance, my main concern is that the sensitivity to policy parameterization itself is not a unique problem of the social dilemma games. It is simply caused by the fact that with certain policy parameterization, even when all opponents' policy are fixed, vanilla policy gradient could fail to improve the agent's policy. If this is the case, then it should not be considered a failure of the LOLA algorithm. And this will likely limit the significance of this paper's contribution.

Additionally, while POLA demonstrates better empirical performance than LOLA in IPD and the Coin game, there are some points that I hope can be explained better either theoretically or empirically:

- The differences and connections between POLA and ideal LOLA. POLA can be viewed as the LOLA algorithm that assumes all opponents update their policies by proximal gradients, not by vanilla gradients. Will this affects POLA's ability to find reciprocity-based cooperation policy?

- In addition to the hyperparameters used by LOLA (the opponent learning rate and the self learning rate), POLA additionally includes the penalty rate \beta_in and \beta_out. Are these additional hyperparameters both crucial to the performance of POLA?
questions: See questions in the strengths and weaknesses section.
limitations: The authors addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes POLA, an extension to LOLA. LOLA is an opponent-shaping technique that enables RL agents to converge to conditional cooperation in mixed environments. A downside of LOLA is that it assumes specific agent parameterization and its updates depend on this parameterization. Authors empirically demonstrate that different parameterization can result in different equilibrium. POLA mitigates this downside by shifting the opponent shaping from the parameter space to the policy space. Empirically, POLA outperforms LOLA (in frequency of converging to tit-for-tat) in iterated prisoner dilemma with history, as well as in coin game (a temporally-extended social dilemma).
strengths_and_weaknesses: Overall, the paper presents a clear and solid improvement of a relevant algorithm. This impovement is a bit marginal, and the algorithm has some practical limitations (only applicable to 2 agents, requires access to environment model or transition dynamics), but still the contribution is significant. I think LOLA-based approaches are currently quite unique in trying to converge to tit-for-tat rather than unconditional cooperation that most existing methods aim for. So improvements of this approach are welcome.

The presentation has no major downsides. The appropriate literature is cited. The issue with LOLA solved by POLA is clearly formulated and validated. The algorithm is comptetently formalised and practical implementations are proposed.

The experiments are adequate, but I have a couple of concerns and propositions that I describe below.

**Minor concerns**

- In sec 2.2 it would also make sense to cite a multitude of approaches to unconditional cooperation that are not robust against defection (without details on how they work), e.g. inequity aversion, social influence, etc. This would emphasize uniqueness of LOLA.

- Lines 160-164: makes sense that POLA does not require assuming opponent’s architecture. An ablation experiment could be added to test it where the opponent’s architecture is varied (like width or depth of the opponent's network).

- Lines 232-234: why are these probabilities used to define TFT? It seems to me that having 0.64 probability to cooperate if the opponent last defected is still far from TFT. May be provide results from a different perspective by reporting in the Appendix average probabilities to cooperate in the five states for each algorithm. This way, the algorithms could be compared numerically without arbitrary thresholds.

- The coin game, while harder than IPD, is still only 3x3. Ideally, the algorithms would also be compared in environments with bigger maps (may be downscaled CleanUp or Harvest).

- In figures, linestyles and markers should be used so that all curves could be discerned without relying on colors.

questions: Sec 3.1: is there an explanation of why these Q^i are used? Why does LOLA fail (or why does sensitivity to parameterezation results in different equilibrium)? Does LOLA fail on other Q^i that only change basis?
limitations: Limitations and societal impact are adequately addressed.
review_title: null

Meta-review: metareview: I read the paper, review, and responses. I'm not an expert in this sub-area. I'm consider it an incremental work with borderline accept/reject evaluation based on the paper and reviews. OK with an accept.
confidence: Less certain
Decision: Accept