Proximal Learning With Opponent-Learning Awareness
Authors: Stephen Zhao
Chris Lu
Roger Baker Grosse
Jakob Nicolaus Foerster
Venue: NeurIPS 2022 Accept (2022)
Keywords: multi-agent reinforcement learning
LOLA
opponent shaping
proximal point methods

Abstract: Learning With Opponent-Learning Awareness (LOLA) (Foerster et al. [2018a]) is a multi-agent reinforcement learning algorithm that typically learns reciprocity-based cooperation in partially competitive environments. However, LOLA often fails to learn such behaviour on more complex policy spaces parameterized by neural networks, partly because the update rule is sensitive to the policy parameterization. This problem is especially pronounced in the opponent modeling setting, where the opponent's policy is unknown and must be inferred from observations; in such settings, LOLA is ill-specified because behaviorally equivalent opponent policies can result in non-equivalent updates. To address this shortcoming, we reinterpret LOLA as approximating a proximal operator, and then derive a new algorithm, proximal LOLA (POLA), which uses the proximal formulation directly. Unlike LOLA, the POLA updates are parameterization invariant, in the sense that when the proximal objective has a unique optimum, behaviorally equivalent policies result in behaviorally equivalent updates. We then present practical approximations to the ideal POLA update, which we evaluate in several partially competitive environments with function approximation and opponent modeling. This empirically demonstrates that POLA achieves reciprocity-based cooperation more reliably than LOLA.


Proximal Learning With Opponent-Learning Awareness
Stephen Zhao
Chris Lu FLAIR, University of Oxford
University of Toronto and Vector Institute stephen. zhao@mail www.orto.oooooom ca
christopher lu@exeter. ox. ac.
Roger Grosse
Jakob Foerster
University of Toronto and Vector Institute rgrosselcs toronto.comm edu
FLAIR, University of Oxford jakob. foersterennt....g...ooo ox ac uk
Abstract
Learning With Opponent-Leamingg Awareness (LOLA) (Foerster al. (018a] multi-agent reinforcement learning algorithm that typically learns reciprocity-basee cooperation in partially competitive environments. However, LOLA often fails to learn such behaviour on more complex policy spaces parameterized by neural networks, partly because the update rule is sensitive to the policy parameterization. This problem especially pronounced in the opponent modeling setting, where the opponent's policy is unknown and must be inferred from observations; in such settings, LOLA is ill-specified because behaviourally equivalent opponent policies can result in non-equivalent updates. To address this shortcoming, we reinterpret LOLA as approximating a proximal operator, and then derive a new algorithm, proximal LOLA (POLA), which uses the proximal formulation directly. Unlike LOLA, the POLA updates are parameterization invariant, in the sense that when the proximal objective has a unique optimum, behaviourally equivalent policies result in behaviourally equivalent updates. We then present practical approximations the ideal POLA update, which we evaluate in several partially competitive environments with function approximation and opponent modeling. This empirically demonstrates that POLA achieves reciprocity- based cooperation more reliably than LOLA.
Introduction
As autonomous learning agents become more integrated into society, increasingly important to ensure these agents' interactions produce socially beneficial outcomes, i.e. those with high total reward. One step in this direction ensuring agents are able to navigate social dilemmas [Dawes] 1980]. Foerster et al. [2018a] showed that simple applications of independent reinforcement learning (RL) to social dilemmas usually result in suboptimal outcomes from a social welfare perspective. To address this, they introduce Learning With Epprnint-Learning Awareness (LOLA), which actively shapes the learning step other agents. LOLA with tabular policies learns tit-for-tat (TFT), i.c reciprocity-based cooperation, in the iterated prisoner's dilemma (IPD), 2-agent social dilemma.
However, for the same IPD setting, we show that LOLA with policies parameterized by neural networks often converges to the Pareto suboptimal equilibrium of unconditional defection. This shows that the learning outcome for LOLA is highly dependent on policy parameterization, one factor that makes LOLA difficult to scale to higher dimensional settings. This problem is especially pronounced in the opponent modeling setting, where the opponent's 's policy unknown and must be
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

inferred from observations; in such settings, LOLA is ill-specified because behaviourally equivalent opponent policies can result non-equivalent updates and hence learning outcomes.
Agent Updated Pollcy (UDUA)
Aogg 2 Digeooo Polio (POL))
0.50
XP
0.50
DD CD ONN
00
MCC JNN
CD (NN
c INN
CD (NN
aD (NN
CCC INN
CC
CCN
CC 2)
0.35
0.10
Agent 1 Updated Policy (UOLA)
Agent Updated Policy (POLA)
Figure 1: In the IPD (Section 4.1), initialize two neural networks using the same architecture with different weights that produce the same policy, and plot the probability of cooperation in 4 states (DD, DC, CD, CC) after a single update of L.OLA and outer POLA (Section 3.3) Left: using LOLA:; comparing circles to crosses, the different policy parameterizations result in very different updated
policies despite the same starting policies and hyperparameters. Right: using outer POLA; the two updated policies are much more similar.
To address these issues, we build on ideas from proximal algorithms [Parikh and Boyd [014), reinterpreting LOLA as 1-step approximation of such proximal operator, and then devise new algorithm, proximal LOLA (POLA), which uses this proximal formulation directly. POLA replaces the gradient updates in L.OLA with proximal operators based on a proximity penalty in policy space, which makes POLA invariant to policy parameterization. When the proximal objective has a unique optimum, behaviourally equivalent policies result in behaviourally equivalent updates.
Solving for the exact POLA update is usually intractable, so we also provide practical algorithms for settings with and without environment rollouts that rely on approximate versions of the POLA update, We show empirically that our algorithms learn reciprocity-bait -based cooperation with significantly higher probability than LOLA in several settings with with function approximation, even when combined with opponent modeling.
Figure illustrates our core idea: LOLA is sensitive policy parameterization, whereas POLA is not. We compare the same policy parameterized by two different sets of weights in the same neural network architecture in the IPD) showing that our approximation the POLA update is largely parameterization independent, while the LOLA update varies depending on parameterization.
For reproducibility, code is available at:
com wttp://w....omiiileeeeeeerer///e
1.1 Summary of Motivation, Contributions and Outline of Paper Structure
Reciprocity-based cooperation is desired learning outcome in partially competitive environments (Section 2.2). LOLA typically learns such behaviour in simple policy spaces, but not on more complex policy spaces parameterized by neural networks. Our goal is to more reliably learn reciprocity-bate based cooperation in such policy spaces.
Motivation:
Contributions:
We identify and demonstrate previously unknown problem with LOLA that helps explain the aforementioned issue: IOLA is sensitive to policy parameterization (Section 3.1).
To address this sensitivity, we conceptually reinterpret LOLA as approximating a proximal operator, and derive new algorithm, ideal POLA, which uses the proximal formulation directly (Section 3.2
Ideal POLA updates are invariant to policy parameterization, but usually intractable in practice. So,
we develop approximations to the ideal POLA update for use in practice (Sections 3.3 3.4).
We demonstrate that our approximations more reliably learn reciprocity-based cooperation than LOLA several partially competitive environments (Section 4.T). including with function approxi- mation and opponent modeling (Sections 4.2 4.3)

---

2 Background
2.1 Reinforcement Learning
We consider the standard fully-observable reinforcement learning and Markov Game formulation:
An N-agent fully observable Markov Game M is defined as the tuple M
R,,)) where R, 7),
the state space, A = {AI,,.. AN is a set of action spaces, where A, for i E [1,..., N denotes the action space for agent i, T S x Ai X X AN + P(S) is transition function mapping from states and actions to probability distribution over next states, R = {R1,.., Rx) J is a set reward x AN R denotes the reward function for agent i, and y E R
functions, where R² S A is a discount factor. Each agent acts according to a policy Tg. : P(A.). with parameters 8¹.
Let 0 be the concatenation of each of the individual 6 (so for N = 2. {0',02) and be the concatenation of each of the individual igg.. Each agent's objective is maximize its discounted expected return: ('To (S))(a) Eio where To is joint action
{a1,...,ay E Ai x X An drawn from the policies To given the current state 8 E S, P(S) the probability distribution over states given the current policies and transition function, ri denotes the reward achieved by agent at time step t defined by the reward function R, and T N is the total time horizon or episode length. Throughout this paper, we assume all states are visited with non-zero probability. This is true in practice with stochastic policies, which we use throughout our experiments.
Let L' (TT) = - Each agent's objective is equivalently formulated as minimizing (TTe)). We use this as better aligns with standard optimization frameworks. For ease of exposition throughout this paper we consider the N = 2 case and consider updates from the perspective of agent 1. Updates for agent follow the same structure, but with agents and swapped.
2.2 Reciprocity-Base Cooperation and the IPD
Reciprocity- cooperation refers to cooperation iff others cooperate. Unlike unconditional cooperation, this encourages other self-interested learning agents to cooperate. One knol- example is the tit-for-tat (TFT) strategy in the IPD [Axelrod and Hamilton 1981]. At each time step in the IPD, agents cooperate (C) or defect (D). Defecting always provides higher individual reward compared to cooperating for the given time step, but both agents are better off when both cooperate than when both defect. The game is played repeatedly with a low probability of termination at each time step, which is modeled by the discount factor Y in RL. TFT agents begin by cooperating, then cooperate iff the opponent cooperated at the previous time step. Against TFT agent, the best strategy for a self-interested agent is to cooperate at every time step. Other examples of reciprocity- based cooperation include contributing reciprocating by collecting only resources that do not harm others (such as in Section 4.3) Crucially, unconditional cooperation, for example in the single step prisoner's dilemma or against defecting opponents in the IPD, is usually a dominated strategy and thus not a desired learning outcome. Thus, we do not compare against works that modify the learning objective and in principle learn unconditional cooperation such as Hughes et al. 2018 Wang et al. [2018] Jaques et al. (2019)
common good when others contribute (to avoid punishment), or
McKee et al. 22220]
2.3 Learning with Opponent-Learning Awareness (LOLA)
LOLA [Foerster al. 2018a introduces opponent shaping via a gradient based approach. Instead of optimizing for L (TTT,, TT²) i.c. the loss for agent 1 given the policy parameters (0), (2), agent 1 optimizes for L (Toi, , -o------0 (0))) the loss for agent after agent updates its policy with one naive learning (NL) gradient step A = Ve
ge L2 (Taa Tga),, where 17 the opponent's learning rate. Importantly, A62 is treated as function of 0¹, and LOLA differentiates through the update A (0+) when agent 1 optimizes for (TTT, TT²) Appendix A. provides more details, including pseudo-code for LOLA. Foerster et al. [018a] also provide a policy gradient based formulation for use with environment rollouts when the loss cannot be calculated analytically, and use opponent modeling to avoid needing access to the opponent's policy parameters.
3

---

In the IPD with tabular policies, LOLA agents learn a TFT strategy, with high probability cooperating when the other agent cooperates and defecting when the other defects [Foerster et al. 2018a]. Thus, LOLA showed that with the appropriate learning rule self-interested agents can learn policies that lead to socially optimal outcomes the IPD.
2.4 Proximal Point Method
Following Parikh and Boyd 2014), define the proximal operator proxas _ + R" of a closed proper convex function f as:
x arg min (f(z) + 2 ||0 2012 2A
x
where |1- l22 the Euclidean (L2) norm and is hyperparameter. f is differentiable, its first-order approximation near v is:
(x) = f(v) + (z(o)"(x-o) o)
The proximal operator of the first-order approximation is:
prox go (v) arg min (S()) (f()+ u) 2x |1 u112
AVS(o)
which is a standard gradient step on the original function f(u) with step size A.
The proximal point method starts with an iterate Zp. then for each time step t E f1, 2.. calculates a new iterate H, = ( Gradient descent is equivalent to using the proximal point method with first-order approximation of f.
3 Proximal LOLA (POLA)
In this section, we first explore how LOLA is sensitive to different types of changes in policy parameterization (Section 3.T). Next, introduce ideal POLA (Section 3.2). method invariant
to all such changes. Lastly, we present approximations to POLA and resulting practical algorithms (Sections 3.3 3.4).
3.1 Sensitivity to Policy Parameterization
LOLA is sensitive to policy parameterization, as it is defined in terms of (Euclidean) gradients, which are not invariant to smooth transformations of the parameters. Specifically, parameterization affects not only the convergence rate, but also which equilibrium is reached, as we illustrate with a simple toy example. Consider the case of tabular policies with one time step of memory for the IPD. There are five possible states: DD (both agents last defected). DC (agent 1 defected while agent cooperated), CD (agent cooperated, agent 2 defected), CC (both agents last cooperated), and the starting state. For each agent E {1,2), 8 E Rd parameterizes a tabular policy over these 5 states, so that To (s) sigmoid (0') [s). where u[s] denotes the element vector v corresponding to state s. apply the invertible transformation 20 where:
To illustrate the dependence on parameterization,
0 20
2000
01 -200
0
000
00
Q
2 22000
000
00 -2
2001
Thus, agent i's policy is now To oi (s) sigmoid (100))08e
Definition 3.1. For policies Toin, Rgit, we say Tgin Tgt when for all s E S, To:n (8) Tgi
(gi) (s).
The transformation matrices Q² Q² are non-singular and represent changes of basis. Thus, for any policy wa., there exists some 8 such that Tg = Rg. Despite this, LOLA fails to learn TFT in this transformed policy space (Figure 3). We also observe this issue when comparing tabular LOLA to LOLA with policies parameterized by neural networks Figure changes policy representation materially affect LOLA. Relatedly, LOLA is sensitive to different policy parameterizations with the same policy representation, as Figure (left) shows. In the next subsection, we propose an algorithm that addresses these issues.

---

3.2 Ideal POLA
In this section, we first formalize the desired invariance property. Next, we introduce an idealized (but impractical) version of POLA, and show that it achieves the desired invariance property.
Definition 3.2. An update rule u I" x I + R that updates policy parameters 0* E using auxiliary information 0 E IRm is invariant to policy parameterization when for any 0 E RR", 0 IR", 02 E IR"", 0 E R.' such that Tgt. = Tyio and T2za = Tgan, Ta(Ole _0-)) = Tu( ,02)-. short, if the original policies were the same, are the new policies (but not necessarily the new policy parameters).
To achieve invariance to policy parameterization, we introduce our idealized version of proximal LOLA (ideal POLA). Similarly to PPO [Schulman al. [017], each player adjusts their policy to increase the probability of highly rewarded states while penalizing the distance moved in policy space. Crucially, each player also assumes the other player updates their parameters through such a proximal update. More formally, at each policy update, agent I solves for the following 0":
0²(0" 02) = arg min (L (TTa, Toor(oir,00)) + DounD(r [[Tr)))))))))
where (TT ( is shorthand for general divergence defined policies; specific choices are given subsequent sections. For 6² in Equation [
we choose the following proximal update:
0² (0"0 0²) arr min (TTii, + B ((T0e) [|TO="))
(2)
62M
For the above equations to be well defined, the arg min must be unique; assume this in all our theoretical analysis. If different parameters can produce the same policy, or multiple different policies have the same total divergence and loss, then the arg min is non-unique. However, non-unique solutions is not an issue in practice, as our algorithms approximate arg min operations with multiple gradient updates.
Theorem 3.3. The POLA update rule u()',02) ('(0', (2), where ' is defined based on Equa-
tions] 7 and 2
is invariant to policy parameterization.
The proof is in Appendix A.2. In short, this follows from the fact that all terms in the arg min in Equations] and are functions of policies, and not directly dependent on policy parameters.
This invariance also is a major advantage in opponent modeling settings, where agents cannot directly access the policy parameters of other agents, and must infer policies from observations. LOLA is ill-specified in such settings; the LOLA update varies depending on the assumed parameterization of opponents' policies. Conversely, POLA updates are invariant policy parameterization, any parameterization can be chosen for the opponent model.
To clarify the relation between LOLA and POLA, LOLA is a version of POLA that uses first-order approximations to all objectives and an L2 penalty on policy parameters rather than a divergence over policies. This follows from gradient descent being equivalent to the proximal point method with first order approximations (Section) 2.4); we provide a full proof in Appendix A.3.
Exactly solving Equations I and 2 is usually intractable, so in the following Sections 3.3 and 3.4
we
formulate practical algorithms that approximate the POLA update.
3.3 Outer POLA
To approximate ideal POLA, we use a first order approximation to agent 2's objective, which is equivalent agent 2 taking gradient step (see Theorem A.3 for details). That is, instead of finding 0²r via Equation 2 we use 2²" = A02. Agent thus solves for:
(0(00_ 02) arg min (T. (TTIi,, Th- (62(81))) + Boa ((rg [|=
(3)
Solving the arg min above exactly is usually intractable. For practical algorithm, we differentiate through agent 2's gradient steps with unrolling, like in LOLA, and repeatedly iterate with gradient steps on agent I's proximal objective until a fixed point is found; Algorithm [ shows pseudo- code. We choose (T (igg:l/ E.)U()) (D 1 (TTa (s)]l (s))), where US) denotes uniform distribution over states, as this most analogous an 1.2 distance on tabular policies, and we only test outer POLA on settings where tabular policies can be used.
5

---

Algorithm Outer POLA 2-agent formulation: update for agent
Input: Policy parameters 0*, 00,, proximal step size at, learning rate 1), penalty strength Bout
Make copy: 02" t-0 0²
repeat
8 + 82 - 1/V oL (Tgie,
( Tpp)
8t" +- gl" - a1 V 01 (L' (Tain,, Tggu)) + Sour (B (s)] Tazn To)n()))))
until 61" has converged to a fixed point
Output: 02"
3.4 POLA-DICE
Outer POLA assumes we can calculate the exact loss, but often we need to estimate the loss with samples from environment rollouts. For these cases, introduce policy gradient version of POLA adapted to work with DiCE [Foerster et al. 2018b]. DiCE is a sample-based estimator that makes it easy to estimate higher order derivatives using backprop. More details about DiCE and its combination with LOLA are in the Appendix (A.4, A.5) and in Foerster et al. 2018b].
POLA-DiCE approximates the arg min in ideal POLA (Equations [ and 2 with fixed num- ber of gradient steps on the proximal objectives, where steps on the outer objective ① dif- ferentiate through the unrolled steps the inner objective (2) We choose ( Tgue)
(DLs(DKL((F (s)l Tgen (s))], and approximate the expectation under the true state visitation with an average over states visited during rollouts. SET denotes the states from set of roll- outs with T time steps: SET (TT Aggn(1851
E {T), where 8, is the state time step t. [DRL ( (8) ((TT))( (8)) denotes our sample based approximation {st
Σ
SESET
As in LOLA-DiCE, rollouts are done in a simulator, and used in For our experiments, we assume full knowledge of transition dynamics, but in principle an environment model could be used instead. Algorithm provides pseudo-code for POLA-DiCE.
to ES DEL (Tg (s)
(8))]-
the DiCE loss L ((TT),TTa)) (Ta))"
Algorithm 2 POLA-DiCE 2-agent formulation: update for agent
Input: Policy parameters 0², 0 learning rates on, 02, penalty hyperparameters 3in, Sout, number of outer steps M and inner steps K
Initialize: 0td <- 01
for m in 1...A do
Initialize: 62D + 02
for in 1...K do
Rollout trajectories with states in
S using policies (Tai Tgan) Pin (TTe,, mgan (122))
62d + 02" 02 V g2n L2
(
end for
Rollout trajectories with states ST using policies (Toir, T020) ( + Bour Tgin sout
@tn + 0"" V (C)
end for Output: 8
For sufficiently large numbers of inner steps K and outer steps M, and sufficiently small learning rates, POLA-DiCE iterates until a fixed point found. Unfortunately, iterating to convergence often requires a very large amount of rollouts (and memory for the inner steps). When M = 1 and 3m, Bom. = 0, POLA-DiCE is equivalent to LOLA-DiCE [Forster et al., 2018b].
Without opponent modeling, LOLA-DiCE and POLA-DiCE directly access the other agent's policy parameters, using those in simulator rollouts for policy updates. the opponent modeling (OM) setting, agents can only access actions taken by the other agent, from real environment rollouts. For policy updates, agents must use learned policy models in simulator rollouts. We learn policy models by treating observed actions targets in supervised classification setting, in behaviour cloning

---

[Ross et al. 2011]. Figure 2 illlutrates the process at each time step; we keep policy models from previous iterations and update merementally on newly observed actions.
No OM:
OM:
Figure 2: Illustration of the training process each time step for LOLA-DiCE and POLA-DiCE. Without opponent modeling, each agent directly uses the other agent's policy parameters simulator rollouts for policy updates. With opponent modeling, each agent first learns policy model of the opponent based on observed actions from real environment rollouts. Agents then use the learned models in simulator rollouts for policy updates. We assume known dynamics, environment model needs to be learned for simulator rollouts.
Experiments
To investigate how useful our approximations to ideal POLA are, we run experiments to answer the following questions: 1) Does outer POLA learn seiprocity-basee cooperation more consistently than LOLA, across a variety of policy parameterizations, in the IPD with one-step memory (Section 4.1 cooperation more consistently than LOLA-DiCE in settings that require function approximation and rollouts (IPD with full history (Section 4.2) and coin game (Section 4.3) ) so, do these results still hold when using opponent modeling?
2) Does POLA-DiCE learn reciprocity-
4.1 One-Step Memory IPD
With one-step memory, the observations are the actions by both agents at the previous time step. In this setting, we can use the exact loss in gradient updates (see Appendix B. for details).
To investigate behaviour across settings that vary the difficulty of learning reciprocity- based coopera- tion, we introduce a cooperation factor f E R, which determines the reward for cooperating relative to defecting. At each time step,
the number of agents who cooperated. Each agent's reward c f/2 1 Jagent contributed]. This is a specific instance of the contribution game from Barbosa (et al. 2020], with two agents. <
satisfies social dilemma characteristics, where defecting always provides higher individual reward, but two cooperators are both better off than two defectors.
For more details on the problem setup, see Appendices B.1.2 and B.1.3.
Figure Souure3comppnn compares LOLA and outer POLA (Section 3.3) using various f and policy parameterizations. To provide succinet graphical representation of how well agents learn reciprocity- based cooperation, we test how often agents find TFT. We consider agents cooperating with each other with high probability, achieving average reward 80% of the socially optimal, but both cooperating with probability 0.65 if the opponent last defected, to have found TFT. These thresholds are somewhat arbitrary; Appendix 1.5 provides detailed policy probabilities that support our conclusions without such thresholds.
We reproduce Foerster Foorossereeereeerrrre et al. (2018a) 8a] 's result that naive learning converges to unconditional defection, while LOLA using tabular policies finds TFT. However, changes in policy parameterization greatly hinder LOLA's ability find TFT. contrast, outer POLA finds TFT even with function approxima- tion, and finds TFT significantly more often than LOLA in the pre-conditioned setting described in Appendix 1.1.4 further discusses hyperparameter settings.
Section 3.1
Qualitatively, Figure shows that in the IPD with one-step memory, f f= 1.33, and neural network parameterized policies, outer POLA closely approximates the invariance provided by ideal POLA.
To demonstrate that POLAA allows for any choice of opponent model, Appendix B. provides further experiments in the IPD with opponent modeling, using version of POLA similar to POLA-DiCE
7

---

% of Runs finding TF1
50%
0%
0.90
1.00
Cooparation Factor
-LOLA (pre-condition)
Outer POLA (Tabular)
LOLA (Tabular)
oodee Outer POLA (NN)
Outer POLA pre-condition)
Figure 3: Comparison of LOLA and outer POLA in the one-step memory IPD with various f. plotting the percentage of 20 runs that find TFT. "NN" denotes policies parameterized by neural networks. "Pre-condition" denotes the setting Section 3.1 Given hyperparameter tuning, tabular LOLA always finds TFT (for f > 1), whereas LOLA with function approximation fails on lower f. In the pre-conditioned setting, LOLA completely fails to find TFT. In contrast, outer POLA finds TFT regardless whether the policy is tabular or a neural network, and retains most of its performance in the rreccondttooned setting. As sanity checks, naive learning (1 0) always fails to find TFT, and all algorithms always defect for < 1.
4.2
Full History IPD
Next, we relax the assumption that agents are limited to one-step memory and instead consider policies that condition the entire history, which makes using function approximation and rollouts necessary. We parameterize policies using GRUs [Cho et al. 2014) and test whether POLA-DiCE still learns reciprocity-ber -based cooperation within this much larger policy space. We use f 1.33 for the reward structure. For more details on the problem setting, policy parameterization, and hyperparameters,
Appendix B.2. Figure shows results this setting. LOLAA agents sometimes learn reciprocity-wwt cooperation but often learn to always defect, achieving low scores against each other average. POLA agents learn reciprocity-base -base cooperation much more consistently, almost always cooperating with each other but defecting with high probability if the opponent always defects. Furthermore, opponent modeling works well with POLA; POLA agents behave similarly using opponent modeling instead of accessing policy parameters directly.
5ome Messgg Agento thrkot zergle
Average Score vs Each Other
Scor (hesrrce Saperte Meat Lengatto
Average Score vs Always Defect
0.50
0.30
-0.05
0.70
0.2
POLA-DIC
Detect
005
LOLA DKCE
RCA 6-11
-022
0.2
Alsays Cooperate
-0.30
Alsays Defec:
19300
L.Ld Frundeer of Outic Sieps
Mall Nombo of Oste Stare
Figure Comparison LOLA-DiCE and POLA-DiCE on the IPD with GRU parameterized policies that condition the full action history. Left: POLA-DiCE agents learn to cooperate with each other with high probability, achieving close to the socially optimal reward (always cooperate), whereas LOLA agents cooperate with each other much less consistently, often learning to always defect. Right: we test the learned policies against a hard-coded rule that always defects. POLA-DiCE agents defect against such opponent with high probability, achieving close to the optimal reward of 0. showing POLA agents cooperate only when the other agent reciprocates. POLA-OM (POLA-DiCE A-DCE with opponent modeling) agents show similar behaviour to POLA-DiCE agents. All results averaged over 10 random seeds with 95% confidence intervals shown.
4.3 Coin Game
To investigate the scalability of POLA-DiCE under higher dimensional observations and function approximation, we test LOLA-DiCE and POLA-DiCE in the coin game setting from Lerer and |Peysakhovich| [2017]. The coin game consists of a 3x3 grid in which two agents, red and blue, move around and collect coins. There is always one coin, coloured either red or blue, which spawns with

---

the other colour after being collected. Collecting any coin grants reward of I; collecting a coin with colour different from the collecting agent gives the other agent reward. The coin game embeds a temporally extended social dilemma; if agents defect by trying to pick up all coins, both agents get 0 total reward expectation, whereas if agents cooperate by only picking up coins of their own colour, they achieve positive expected average reward (maximum: 3 per time step). Appendix B.3 provides more detail. We again parameterize the agents' policies with GRUs [Cho et al., 2014].
Figure 5 shows that POLA-DiCE agents learn to cooperate with higher probability than LOLA-DiCE agents. picking up larger proportion of their own coins, and again do not naively cooperate. cooperation, but slightly POLA-DiCK agents with opponent modeling also learn reciprocity-basee less so than with direct access to policy parameters, likely due to the noise from opponent modeling.
Proportion of Same Coirs Ricked Up
0.33
Average Score VS Each Other
Syare Mene over Aagenggagttt. and Phllbot uangltt
Average Score vs Always Defect
0.03
-0.2
0:25
0..5
222
-60
Cooc (Bweraye
Detec
-04
POLA DICE
LOUL- DICE
2.05
PCl 6-W
-055
Altrays Cocparate
0.0-
Altrays Csfect
30001
Total hunberr Outer Steps
Rnial Nari bey
of Outer M0e5s
kotal hurtburerr of Ourer Steps
Figure 5: Comparison of LOLA-DiCE and POLA-DiCE on the coin game. Left: number of coins collected where the coin's 's colour matched the agent's 's colour, divided by the total number of coins collected. POLA agents cooperate more than LOLA agents, collecting a larger proportion of their own coloured coins. Middle: average score for the agents against each other. POLA-DiCE agents achieve close to the maximum cooperative reward. Right: we test the learned policies against a hard-coded rule that always defects by taking the shortest path to the coin regardless of colour. POLA agents defect, achieving scores close to the optimal of O against such an opponent, again showing POLAA agents cooperate based reciprocity. LOLA agents quickly learn always defect; subsequent training is highly unstable.
Related Work
POLA-DiCE (Section 3.4 can alternatively be motivated replacing the policy gradient update inherent in LOLA-DICE with a proximal-style update like that of TRPO [Schulman et al. 2015a or the adaptive KI.-penalty version of PPO (Schulman et al.) (017).
There are other extensions to LOLA that address problems orthogonal to policy parameterization sensitivity. In concurrent work. Willi et al. [2022) highlight that LOLA is inconsistent in self-play; the update each player assumes the opponent makes different from their actual update. They propose COLA, learning update rules A and A0 that are consistent with each other. However, their notion of consistency is gradient-based and thus not invariant to policy parameterization. whereas POLA is not consistent. SOS [etcher et al., 20188) which combines the advantages of LOLA and LhokA Zhang and Lesser 2010), is also not invariant to policy parameterization.
Al-Shedivat et al. 20188 formulate a RI. framework with policy gradient theorem, and apply PPO as their optimization algorithm; this one instance of combining proximal style operators with reinforcement learning style policy gradients for optimization with higher order gradients. However, they focus on adaptation rather than explicitly modeling and shaping opponent behaviour.
[Wen et al., 2021 sets up metagame between the original policies and new policies calculated with independent TRPO updates for each agent, solving for Nash equilibrium in the metagame at each step. MATRL finds a best response without actively shaping its opponents' learning. so it is similar to LookAhead [Zhang and Lesser 2010) (which results unconditional defection) rather than LOLA.
MATRL
M-FOS [Lu al. 2022 sets up a metagame where each meta-step is full environment rollout, and the meta-reward at each step is the cumulative discounted return of the rollout. They use model-free
See Appendix B.3 for why the LOLA results cannot be directly compared with Foerster et al. 2018a

---

optimization methods to learn meta veta-policies that shape the opponent's 's behaviour over meta meta-steps. We do not compare against it as it is concurrent work without published code the time of writing.
Badjatiya et al. 2021 introduce status-quo loss, which agents imagine the current situation being repeated for several steps; combined with the usual RL objective, this leads cooperation in social dilemmas. However, their method learns to defect in the one-step memory IPD in the states DC and CD, and thus does not learn reciprocity-baie
cooperation.
Zhao et al. 2021] provide detailed numerical analysis of the function approximation setting for two-player zero-sum Markov Games. Fiez et al. [2021] considers proximal method for minimax optimization. In contrast to both, we consider the general-sum setting.
6 Limitations
Outer POLA and POLA-DiCE approximate ideal POLA, parameterization; Figures [ and show how close the approximation is in the one-step memory IPD. POLA introduces additional hyperparameters compared LOLA (e.g. Bin: Sour and number of outer steps for POLA-DiCE). POLA usually requires many rollouts for each update; future work could explore ways to mitigate this. For example, in Appendix A.8 formulate a version of POLA-DiCE closer to PPO [Schulman et al. 2017]. that repeatedly trains on samples for improved sample efficiency.
completely invariant to policy
7 Conclusion and Future Work
Motivated by making LOLA invariant to policy parameterization, we introduced ideal POLA. Em- pirically, we demonstrated that practical approximations ideal POLA learn reciprocity- cooperation significantly more often than LOLA in several social dilemma settings.
Future work could explore and test: policy parameterization invariant versions of LOLA extensions like COLA [Willi et al. 2022] and SOS [Letcher et al. 2018): ; POLA formulations with improved sample efficiency (e.g. Appendix A.8); the N-agent version of POLA (Appendices A.9, A.10) in N- agent versions of the IPD BBrboosa et al. [020) and larger environments such as Harvest and Cleanup [Hughes al.) 2018); POLA formulations with adaptive penalty parameters 8. Pout' connections to Mirror Learning Kuba et al. 2022]: adapting approximations to proximal point methods such as the extra-gradient method |Mokhtari al. 2020) to work with POLA.
We are excited and hopeful that this work opens the door to scaling LOLA and related algorithms to practical settings that require function approximation and opponent modeling.
Acknowledgements
Resources used in this research were provided, in part, by the Province of Ontario, the Government of Canada, and companies sponsoring the Vector Institute. The Foerster Lab for Al Research (FLAIR) is grateful to the Cooperative Al Foundation for their support. We thank the anonymous NeurIPS reviewers for helpful comments on earlier versions of this paper.
References
Maruan Al-Shedivat, Trapit Bansal, Yura Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. In International Conference Learning Representations, 2018.
Robert Axelrod and William Donald Hamilton. The evolution of cooperation. science, 21 11444)) 1390-1396, 1981.
Pinkesh Badjatiya, Mausoom Sarkar, Nikaash Puri, Jayakumar Subramanian, Abhishek Sinha, Sid- dharth Singh, and Balaji Krishnamurthy. Status-quo policy gradient in multi-agent reinforcement learning, arXiv preprint arXiv:2111.1162,, 2021.
10

---

João Vitor Barbosa, Anna H Reali Costa, Francisco S Melo, Jaime S Sichman, and Francisco C Santos. Emergence of cooperation in n-person dilemmas through actor-critic reinforcement learning. In Adaptive Learning Workshop (ALA), AAMAS, 2020.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprini | 1409. /259, 2014.
Robyn M Dawes. Social dilemmas. Annual review of psychology, 31(1):169- 193, 1980.
Gregory Farquhar, Shimon Whiteson, and Jakob Foerster. Loaded dice: trading off bias and variance in any-order score function estimators for reinforcement learning. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 8151-8162, 2019.
Tanner Fiez, Chi Jin, Praneeth Netrapalli, and Lillian Ratifff. Minimax optimization with smooth algorithmic adversaries. arXiv preprint arv:iv2106. 200.:21006.8888 2021.
Jakob Foerster, Richard Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. In Proceedings of the 17th Interna- tional Conference on Autonomous Agents and MultiAgent Systems, pages 122-130. International Foundation for Autonomous Agents and Multiagent Systems, 2018a.
Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric Xing, and Shimon Whiteson. Dice: The infinitely differentiable monte carlo estimator. In International Conference on Machine Learning, pages 1529-1538, PMLR, 2018b.
Edward Hughes, Joel Z Leibo, Matthew Phillips, Karl Tuyls, Edgar Dueñez-Guzman Antonio Gareía Castañeda, lain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, al. Inequity aversion improves cooperation in intertemporal social dilemmas. In Advances in neural information processing systems, pages 3326-3336, 2018.
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International conference on machine learning, pages 3040-3049. PMLR, 2019.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference on Machine Learning. Citeseer, 2002.
Jakub Grudzien Kuba, Christian Schroeder de Witt, and Jakob Foerster. Mirror learning: unifying framework of policy optimisation. arXiv preprint arXiv:2201.02373 2022.
Adam Lerer and Alexander Peysakhovich. Maintaining cooperation in complex social dilemmas using deep reinforcement learning. arXiv preprint arXiv:1707.01066, 2017.
Alistair Letcher, Jakob Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson. Stable opponent shaping in differentiable games. arXiv preprint 11XXi:1811.08460 2018.
Chris Lu, Timon Willi, Christian Schroeder de Witt, and Jakob Foerster. Model-free opponent shaping. arXiv preprint arXiv:2205.014: 2022.
Kevin R McKee, Ian Gemp, Brian McWilliams, Edgar A DuéñezzGuzmáá Edward Hughes, and Joel Z. Leibo. Social diversity and social preferences mixed-motive reinforcement learning. arXiv preprint arXiv:2002.023335 2020.
Aryan Mokhtari, Asuman Ozdaglar, and Sarath Pattathil. unified analysis of extra-gradient and optimistic gradient methods for saddle point problems: Proximal point approach. In International Conference on Artificial Intelligence and Statistics, pages 1497-1507. PMLR, 2020.
Neal Parikh and Stephen Boyd. Proximal algorithms. Foundations and Trends in optimization, 1(3): 127-239, 2014.
Stéphane Ross, Geoffrey J Gordon, and Andrew Bagnell. No-regret reductions for imitation learning and structured prediction. In In AISTATS. Citeseer, 2011.

---

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint (rXiv:1506002435 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Jane Wang, Edward Hughes, Chrisantha Fernando, Wojciech M Czarnecki, Edgar Duéñez- Guzmán, and Joel Z Leibo. Evolving intrinsic motivations for altruistic behavior. arXiv preprint arXiv: /8/ 1.05931, 2018.
Ying Wen, Hui Chen, Yaodong Yang, Zheng Tian, Minne Li, Xu Chen, and Jun Wang. A game- theoretic approach to -agent trust region optimization. arXiv preprint arXiv:2106.066: 06828, 2021.
Timon Willi, Johannes Treutlein, Alistair Letcher, and Jakob Foerster. Cola: Consistent learning with opponent-learning awareness, arXiv preprint urXiv:2203. 04099, 2022.
Chongjie Zhang and Victor Lesser. Multi-agent learning with policy prediction. In Twenty-fourth AAAl conference on artificial intelligence, 2010.
Yulai Zhao, Yuandong Tian, Jason D Lee, and Simon S Du. Provably efficient policy gradient methods for two- two-player zero-sum markov games. arXiv preprint arXiv:2102.08900,, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Appendix
C.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] some assump- tions like non-zero state visitation probability are stated earlier (e.g. in the background section) and apply throughout.
(b) Did you include complete proofs of all theoretical results? [Yes] in the Appendix, with very brief proof sketches in the main paper.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL.) (Yes] Github repo with code and instructions provided in intro (https:// http///gttttbbtt com SSieenntneeeaaae POLA).
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? (Yes] Either in the main paper or in the Appendix (mostly Appendix B).
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Figures an 5
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] Appendix B.5.
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
12

---

(a) If your work uses existing assets, did you cite the creators? [Yes] Details in Appendix B.4.
(b) Did you mention the license of the assets? [Yes] Details in Appendix B.4.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] Github repo with code provided (https: thub. com Mallent-Zeeba/POO.
(d) Did you discuss whether and how consent obtained from people whose data you'l 're using/curating? [N/A] The only "data" is rollouts from environments.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] The only "data" is rollouts from environments.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable?? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "d3EqX6nkKBz", "reviewer": "Reviewer_kxwY", "summary": "The paper proposes POLA, an extension to LOLA. LOLA is an opponent-shaping technique that enables RL agents to converge to conditional cooperation in mixed environments. A downside of LOLA is that it assumes specific agent parameterization and its updates depend on this parameterization. Authors empirically demonstrate that different parameterization can result in different equilibrium. POLA mitigates this downside by shifting the opponent shaping from the parameter space to the policy space. Empirically, POLA outperforms LOLA (in frequency of converging to tit-for-tat) in iterated prisoner dilemma with history, as well as in coin game (a temporally-extended social dilemma).", "questions": "Sec 3.1: is there an explanation of why these Q^i are used? Why does LOLA fail (or why does sensitivity to parameterezation results in different equilibrium)? Does LOLA fail on other Q^i that only change basis?", "limitations": "Limitations and societal impact are adequately addressed.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper presents a clear and solid improvement of a relevant algorithm. This impovement is a bit marginal, and the algorithm has some practical limitations (only applicable to 2 agents, requires access to environment model or transition dynamics), but still the contribution is significant. I think LOLA-based approaches are currently quite unique in trying to converge to tit-for-tat rather than unconditional cooperation that most existing methods aim for. So improvements of this approach are welcome.\n\nThe presentation has no major downsides. The appropriate literature is cited. The issue with LOLA solved by POLA is clearly formulated and validated. The algorithm is comptetently formalised and practical implementations are proposed.\n\nThe experiments are adequate, but I have a couple of concerns and propositions that I describe below.\n\n**Minor concerns**\n\n- In sec 2.2 it would also make sense to cite a multitude of approaches to unconditional cooperation that are not robust against defection (without details on how they work), e.g. inequity aversion, social influence, etc. This would emphasize uniqueness of LOLA.\n\n- Lines 160-164: makes sense that POLA does not require assuming opponent’s architecture. An ablation experiment could be added to test it where the opponent’s architecture is varied (like width or depth of the opponent's network).\n\n- Lines 232-234: why are these probabilities used to define TFT? It seems to me that having 0.64 probability to cooperate if the opponent last defected is still far from TFT. May be provide results from a different perspective by reporting in the Appendix average probabilities to cooperate in the five states for each algorithm. This way, the algorithms could be compared numerically without arbitrary thresholds.\n\n- The coin game, while harder than IPD, is still only 3x3. Ideally, the algorithms would also be compared in environments with bigger maps (may be downscaled CleanUp or Harvest).\n\n- In figures, linestyles and markers should be used so that all curves could be discerned without relying on colors.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YOdjhNtXkFo", "reviewer": "Reviewer_cSUk", "summary": "The paper proposes POLA, a MARL agent that can learn reciprocal policies in general sum games such as iterative prisoner's dilemma (IPD). The proposed algorithm is largely based on LOLA, a MARL agent that can learn reciprocity-based cooperation in the IPD game. The main contribution of this paper is that under certain parametrization of the policy space, LOLA may fail to learn good policies. The paper highlights that a more ideal learning algorithm needs to be invariant with the policy parameterization. Although it is impractical to implement such update rules, the paper borrows ideas from PPO and proposes a proximal update version of LOLA. The proposed algorithm demonstrates better performance on IPD and the coin game.", "questions": "See questions in the strengths and weaknesses section.", "limitations": "The authors addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main merit of this paper is to use a proximal update objective to improve LOLA. Although proximal policy update has been largely explored in the RL community (e.g., TRPO, PPO), it is not previously applied to LOLA. While the combination of LOLA and proximal method lead to better performance, my main concern is that the sensitivity to policy parameterization itself is not a unique problem of the social dilemma games. It is simply caused by the fact that with certain policy parameterization, even when all opponents' policy are fixed, vanilla policy gradient could fail to improve the agent's policy. If this is the case, then it should not be considered a failure of the LOLA algorithm. And this will likely limit the significance of this paper's contribution.\n\nAdditionally, while POLA demonstrates better empirical performance than LOLA in IPD and the Coin game, there are some points that I hope can be explained better either theoretically or empirically:\n\n- The differences and connections between POLA and ideal LOLA. POLA can be viewed as the LOLA algorithm that assumes all opponents update their policies by proximal gradients, not by vanilla gradients. Will this affects POLA's ability to find reciprocity-based cooperation policy?\n\n- In addition to the hyperparameters used by LOLA (the opponent learning rate and the self learning rate), POLA additionally includes the penalty rate \\beta_in and \\beta_out. Are these additional hyperparameters both crucial to the performance of POLA?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UhoW5as8ts", "reviewer": "Reviewer_ghku", "summary": "This work reinterpret LOLA as approximating a proximal operator, and then derive a new algorithm called Proximal LOLA (POLA) which uses the proximal formulation directly. The updates of the POLA method are parameterization invariant, which means that behaviourally equivalent policies can result in behaviourally equivalent updates.", "questions": "- In the results of Full History IPD experiment (Figure 4), there is a significant degradation in the performance curve of the POLA-DiCE method. Is there any specific analysis and explanation about this phenomenon?", "limitations": "This work is an incremental work based on the LOLA method, simply combined with the Proximal Point Method for improvement, and does not have enough novelty.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- This work provides good insights for improving the LOLA method in partially competitive environments.\n\nWeakness:\n\n- This paper is poorly written and hard to follow.\n- The experiments in this paper are not solid enough. And in Section 4, the author only states the experement settings and the results, without analyzing and explaining the results of the experiment.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept