LOQA: Learning with Opponent Q-Learning Awareness
Authors: Milad Aghajohari
Juan Agustin Duque
Tim Cooijmans
Aaron Courville
Venue: ICLR 2024 poster (2024)
Keywords: multi-agent reinforcement learning
social dilemmas
deep reinforcement learning
Primary area: reinforcement learning
TL;DR: Achieve reciprocity-based cooperative agents via q-learning awareness one order of magnitude faster than previous SOTA(POLA).

Abstract: In various real-world scenarios, interactions among agents often resemble the dynamics of general-sum games, where each agent strives to optimize its own utility. Despite the ubiquitous relevance of such settings, decentralized machine learning algorithms have struggled to find equilibria that maximize individual utility while preserving social welfare. In this paper we introduce Learning with Opponent Q-Learning Awareness (LOQA) , a novel reinforcement learning algorithm tailored to optimizing an agent's individual utility while fostering cooperation among adversaries in partially competitive environments. LOQA assumes that each agent samples actions proportionally to their action-value function Q. Experimental results demonstrate the effectiveness of LOQA at achieving state-of-the-art performance in benchmark scenarios such as the Iterated Prisoner's Dilemma and the Coin Game. LOQA achieves these outcomes with a significantly reduced computational footprint compared to previous works, making it a promising approach for practical multi-agent applications.

Published as a conference paper at ICLR 2024
LOQA: LEARNING WITH OPPONENT Q-LEARNING AWARNNESS
Milad Aghajohari", Juan Agustin Duque', Tim Cooijmans, Aaron Courville University of Montreal & Mila fi Frstname. last name aaaamtmrntraallcca
ABSTRACT
In various real-world scenarios, interactions among agents often resemble the dy- namics of general-sum games, where each agent strives to optimize its own utility. Despite the ubiquitous relevance of such settings, decentralized machine learning algorithms have struggled to find equilibria that maximize individual utility while preserving social welfare. In this paper we introduce Learning with Opponent Q- Learning Awareness (LOQA), a novel, decentralized reinforcement learning algo- rithm tailored to optimizing an agent's individual utility while fostering coopera- tion among adversaries in partially competitive environments. LOQA assumes the opponent samples actions proportionally to their action-value function Q. Exper- imental results demonstrate the effectiveness of LOQA at achieving state-of-the- art performance in benchmark scenarios such as the Iterated Prisoner's 's Dilemma and the Coin Game LOQA achieves these outcomes with significantly reduced computational footprint, making it a promising approach for practical multi-agent applications.
INTRODUCTION
A major difficulty in reinforcement learning (RL) and multi-agent reinforcement learning (MARL) the non-stationary nature of the environment, where the outcome of each agent is determined not only by their own actions but also those of other players von Neumann (1928). This difficulty often results in the failure of traditional algorithms converging to desirable solutions. In the context of general-sum games, independent RI. agents often converge to sub-optimal solutions in the Pareto sense, when each of them seeks to optimize their own utility Foerster et al. (2018b). This situation draws parallels with many real-world scenarios, in which individuals pursuing their own selfish interests leads them to a worse outcome than cooperating with others. Thus one of the objectives of MARL research must be to develop decentralized agents that are able cooperate while avoiding being exploited in partially competitive settings. We call this reciprocity-basee cooperation.
Previous work has resulted in algorithms that train reciprocity cooperative agents by differen- tiating through the opponent's learning step (Foerster et al. 2018h) Letcher et al.) 2021 Zhao et al.) 2022 Willi et al., (2022) or by modeling opponent shaping as a meta meta-game in the space agent poli- cies (Al-Shedivat et al. 2018 Kim et al. [2021 Lu et al. 2022 Cooijmans et al. 2023), However, both of these approaches have important drawbacks with respect to computational efficiency. On one hand, differentiating through even just few of the opponent's 's learning steps, can only be done sequentially and requires building large computation graphs. This is computationally costly when dealing with complex opponent policies. On the other hand, meta-learning defines the problem as a meta-state over the product space of policies of the agent and opponent, and learns meta-policy that maps from the meta-state to the agent's updated policy. The complexity of the problem then scales with the policy parameterization which is usually a neural network with many parameters.
In this paper introduce Learning with Opponent Q-Learning Awareness (LOQA), which stands because it avoids computing gradients w.r.t. optimization steps or learning the dynamics of meta- game, resulting in significantly improved computational efficiency. LOQA performs opponent shap- ing by assuming that the opponent's 's behavior is guided by an internal action-value function Q. This assumption allows LOQA agents to build model of the opponent policy that can be shaped by in-

---

Published as a conference paper at ICLR 2024
fluencing its returns for different actions. Controlling the return by differentiating through stochastic objectives is a key idea in RL and can be done using the REINFORCE estimator Williams (992).
BACKGROUND
We consider general-sum, n- n-player, Markov games, also referred as stochastic games [hhaleyy [1933). Markov games defined by tuple M (N, A. R,?) where S denotes the state space, A := AA' ... x A", is the joint action space all players, P S A + A(S), defines a mapping from every state and joint action probability distribution
states, R {r ..... r")
the set reward functions where each X² S x A + I maps every state and joint action to a scalar return and 7 [0, is the discount factor.
We use the notation and definitions for standard RL algorithms of Agarwal et al. (021). Consider two agents, (agent) and (opponent) that interact in environment with neural network policies n² == x-11:;0 0)), x² == x--1:;²) parameterized by 0 and 0² respectively. We denote т to be trajectory with initial state distribution μ and probability measure Pr. .n²" given by
Pr Pr (T) =( (cols)) so P(1 so. ao, by)
here A denotes the action of the opponent. agent reinforcement learning, each agent seeks optimize their expected discounted return for the agent this is given by:
v (1) : E T P. (T) = E PP
rr' (se, ae, be)
The key observation is that under the definitions above, V is dependent the policy of the oppo- nent through the reward function (st, a, by). V thus differentiable with respect to the parame- ters of the opponent via the REINFORCE estimator (Williams) 1992
8 (n)
E TPPP Pr !
(T(
V Bog x= (belst)
1=0
(1)
E PP
Σ Trrr" b.)
Σ Voalog 7 (bulsk)
kCt
we will rely on this observation to influence the opponent's 's Q value and incentivize reciprocity- based cooperation in partially competitive environments.
RELATED WORK
3.1 OPPONENT SHAPING
Learning with opponent learning awareness (LOLA), (Foerster et al. 2018b) introduces the concept of opponent shaping, i.e. the idea steering the other agent throughout its learning process. LOLA assumes that the opponent is naive learner and attempts
shape it by considering one step in its optimization process. Rather than optimizing the value under the current policies at the current iteration i, (=(0,, 02), LOLA optimizes (" 02 + AOf)) where A0? is learning step of the oppo- nent. A0 is as function that depends on the agent's parameters and that is thus differentiable with respect 8', Since the derivative of function v' (0), 0² + A0?) difficult compute, the authors consider the surrogate value given by its first order Taylor expansion. POLA, (Zhao et al.) 2022) builds an idealized version LOLA that, unlike its predecessor, is invariant the parameterization of the policy. In similar fashion to proximal policy optimization (PPO) (Schulman et al.) 2017), each agent increases the probability of actions that increase their expected return, while trying to minimize the (2 distance between the updated policy and the old policy. This combined objective of maximizing the return and minimizing the ln distance in policy space is equivalent to the Proximal Point method, hence the name Proximal-LOLA or POLA. Other modifications the original LOLA algorithm attempt to resolve its shortcomings. Consistent learning with opponent learning aware- ness (COLA), (Willi et al. 2022) attempts resolve the inherent inconsistency of LOLA assuming
2

---

Published as a conference paper at ICLR 2024
Opponent
Cooperate
Defect
Agent
Cooperate Defect
3, 0)
0,
22 2-)
Table 1: Payoff (or reward) matrix for the Prisoner's Dilemma game, where the numbered pairs corresponds to the payoffs of the Agent and the Opponent respectively.
that the other agent naive learner instead of another LOLA agent. Stable opponent shaping (SOS), (Letcher et al. (202T) introduces an interpolation between LOLA and a more stable variant called look ahead, which has strong theoretical convergence guarantees.
3.2
META LEARNING
Other methods have been used to generate agents that have near optimal behavior in social dilemmas. First used by AI-Shedivva al, (2018) for this setting, meta learning redefines the MARL problem as meta-game in the space of policy parameters in an attempt to deal with the non-stationary nature of the environment. In this meta-game, the meta-state is the joint policy, the meta-reward is the return the underlying game, and a meta-action is change to the inner policy (i.e. the policy in the original game). Model free opponent shaping (M-FOS) (Lu et al. 2022) applies policy gradient methods to this meta--ame find strong meta-policy. Meta- Value Learning (Cooijmans et al., 2023) applies value learning to model the long-term effects of policy changes, and uses the gradient of the value improvement direction.
4 SOCIAL DILEMMAS
Social dilemmas type of decision problem where each party's miopic efforts to maximize their own benefit lead to a less favorable outcome compared to when all parties cooperate. Designed primarily thought experiments, these dilemmas demonstrate the trade-off' that often inherent in multi-agent decision-making scenarios. As such, they have been used to model real-life situations in diverse fields such economics, ecology and policy making. One example of such decision problem is the famous Iterated Prisoner's Dilemma (IPD).
The Prisoner's Dilemma (PD) PD is game in which each of the two agents prisoners must decide to either cooperate with one another or defect. The dilemma the prisoners face originates from the reward structure, given in Table With with reward structure, rational agent will choose to defect no matter what the other agent chooses. As a result, both agents become locked in the defect-defect Nash equilibrium, even though they would achieve greater utility by both choosing to cooperate.
Iterated Prisoner's Dilemma (IPD) As the name implies IPD is simply an (infinitely) repeated version of the Prisoner's Dilemma. Unlike standard PD, IPD offers some hope for rational cooper- ative behaviour. Originally popularized by Axelrod (1980), the IPD has been used to model many hypothetical and real-world scenarios. It has also become popular test-bed for MARL algorithms attempting achieve reciprocity based cooperation policies. simple but effective strategy in the IPD is fit-for-tat (TFT), which consists in cooperating at the first turn and copying the opponent's action thereafter.
The Coin Game Initially described in (Lerer & Peysakhovich 2018), the Coin Game grid world environment in which two agents take turns taking coins. At the beginning of each episode coin of a particular color (red or blue), corresponding to that one of the two agents, spawns at random location in the grid. Agents are rewarded for any coin taken, but agent takes the coin corresponding to their color. The reward structure of the Coin Game is designed to incentivize cooperation between agents, as each one would be better off if both take only the coin corresponding to their color. In such way the Coin Game mirrors the IPD, therefore policies that cooperate reciprocally are highly desirable as they achieve better individual and social outcomes.
punished if the other
3

---

Published as a conference paper at ICLR 2024
However unlike IPD, the Coin Game is embedded in non-trivial environment and requires non- trivial policy models. can seen as an extension of IPD towards more complex and realistic scenarios.
METHOD DESCRIPTION
Intuitively, a change in either the agent's or the opponent's policy results in a change in the proba- bility measure over the trajectories that are observed when both them interact an environment. Since the value function of the opponent is expectation said probability measure, it is con- trollable by the agent's policy (and vice versa). LOQA leverages this observation to exert influence over the policy that the opponent will learn.
As an illustration, consider an instance the IPD game where LOQA agent and the opponent are initialized be random agents, i.e. they samples actions from uniform distribution. If the LOQA agent increases its probability of defection after the opponent defects, it implicitly decreases the action-value function of the opponent for defecting. The opponent will then learn this and reduce its probability of defecting. Similarly, if the LOQA agent cooperates after the opponent cooperates, it increases the action- value of cooperation for the opponent. response, the opponent will learn to cooperate more. This reciprocity- recpprocitybbased cooperative behavior is the structure behind tit-for-tat.
5.1 MODELING THE OPPONENT'S POLICY
Let T (bls) = T(bls; refer to the policy of the agent and (b|s) := <(b|s; 0²) refer to the policy of the opponent, which are neural networks with parameters 0z and 02, Similarly, (s) == ( b; x²) denotes the action-value function the opponent, which neural net- work with parameters o²
LOQA relies on a key assumption about the opponent's policy. Similar to Soft Actor Critic (SAC) Haarnoja et al. (018)) the assumption is that the opponents" actions sampled from distribution More formally, at time i, we can write this
that is proportional to its action-value function Q²(.).
assumption as
(st(Q?(ss, Ex exp(("((s, B))
n (b.lst) R
More specifically we approximate Q² with Monte Carlo rollouts T of length T, where every trajec- tory E T. Pr.j starts at state s with the opponent taking action by
Q b,) E [R((s) sc. b.]
ΣΣ x k (5 aus bb)
where rs(s a, b) denotes the opponent's reward at state a after taking action b and the opponent taking action This empirical expectation of the Q function is controllable by the agent using the reinforce estimator
Vo be), = ETNT
Σ k=l+1
k 2
bx
Σ bjjck
Vo म (a,l=;)
The opponent's policy evaluated at state 8; can now be approximated using the Monte Carlo rollout estimate Q² and the action-value function Q² as follows
exp(Q2(ss,,)) b.))
2 (b.lst)) =
b.)) + Ev exp(()(ss,
Notice that assume access to the opponent's real action-valus function Q To have fully decentralized algorithm we can simply replace Q² with the agent's own estimate of the opponent's action-value function. We now integrate these ideas into the Actor-Critic formulation.
4

---

Published as a conference paper at ICLR 2024
Algorithm LOQA
Initialize:
Discount factor ", agent action-value parameters 6 target action-value parameters
Purgets actor parameters 01, opponent action-value parameters o² target action-value parameters 6
larget actor parameters B2 for iteration= ., .....do do
Run policies я and x² for timesteps environment and collect trajectory Lg - 0, L2 0
for t =
... do
l <- 1 HUBER LOSS + 7 (arggee( (8t-1, 0c+1) (8t, ax))
L <- I2+ MUBER_LOSS(T; + 7Q iarger (8c-1: b(+1) (se, oz)) end for
Optimize Lg w.n.t. 6¹ and L3 w.r.t. x² with optimizer of choice
Compute advantage estimates {A]...., A7).
{A} Aff......f. A3}
I. t LOQAAACTON
LOSS(T, T
{A
L + LOQA ACTOR LOSLLOSSTTTTE T², (A........
Optimize L w.r.t 0² and L w.r.t. 0 with optimizer of choice
end for
Algorithm 2 LOQA ACTOR LOSS
Input: Trajectory T of length T, actor policy n², opponent action-value function Q= discount factor 7, advantages (Ai...., A
La +- () for 1,2......
do
0 b)) + Ek=r k- - (sk ak. bk) , b.) and Q "
b ri made differentiable using DiCE
Compute -- using +st,,
(st. by) according to equation La +- La + Ai [log n' (arls)) + log # (brl5e)
end for return: La
5.2 OPPONENT SHAPING
In order shape the opponent behavior, we factor in the opponent's policy approximation #2 as well as the agent's policy n' in the probability measure over trajectories. Adapting the original Actor-Critic formulation (Konda Tsitsiklis. 2000) the joint agent-opponent policy space have:
Vo (p) EE Empp!
Σ A (1,,,
log n ra(aaas)) + log क (belsse)
(
where A' (s (z,bb)) is the advantage of the first agent, and x² is constant w.r.t. 0j. LOQA ap- proximates the opponent's policy using Equation 2 This approximated policy is differentiable with respect agent parameters since it is computed based the opponent's action-value. which is also differentiable (see Equation[]] Consequently, a second term emerges in LOQA's update
a (1) R TPPr / 1
Σ A A'(st, at, Vor (log x (a,(s))) + log # (bels))) t=0
(3)
The first log term comes from the Actor-Critie update and the second log term is shaping compo- nent that pushes the opponent's 's return beneficial direction for agent (in policy space). This second term comes from the assumption that the opponent's policy can be influenced by the agent's s parameters. For a derivation refer to section the appendix.
In practice
DiCE (Foerster et al. 2018a) and loaded-DiCE (Farquhar et al. 2019) the action-value estimate Q² to compute the gradient Vollog #² and reduce its variance. Also, the current trajectory itself is used for Q² estimation. (See appendix

---

Published as a conference paper at ICLR 2024
(Ouns)
Probability Cooperation
State
Figure I: Probability of cooperation of sigmoid LOQA agent at each possible state in the one-step history IPD after 7000 training iterations. LOQA agents' resulting policy is similar to tit-for-tat, policy that cooperates at the first step and copies the previous action of the opponent at subsequent time-steps.
5.3 SELF PLAY AND REPLAY BUFFER OF PREVIOUS AGENTS
Because the environments are symmetric we can use self-play to train single LOQA agent against itself. We also maintain replay buffer and, for each optimization step (which requires generating environment rollouts), we sample uniformly from previously encountered agents. This increases the diversity of the opponents the agent faces in the training. The replay buffer has certain capacity and receives a new agent every rath iteration where n a hyperparameter.
6 EXPERIMENTS
We consider two general-sum environments to evaluate LOQA against the current state-of-the-arr, namely, the Iterated Prisoner's Dilemma (IPD) and the Coin Game. We compare with POLA and M-FOS, the only methods the best of our knowledge that generate reciprocity- cooperative policies in the Coin Game
6.1 ITERATED PRISONER 'S DILEMMA
We train an agent consisting of sigmoid over logits for each possible state in the one-step history IPD. There are possible states in this configuration, namely START (the starting state), CC, CD, DC and DD where C stands for cooperation and D stands for defection. The training is done for 4500 iterations (approximately 15 minutes on an Nvidia A100 gpu) using a batch size of 2048. We empirically observe that LOQA agent is able to reach tit-for-tat like policy shown by looking at the probability of cooperation at each state. We helieve that the probabilities are not fully saturated for reasons. First, the normalization policy approximation makes it numerically impossible to reach a probability of one for either action. Second, observed that after some time the trajectories become homogeneous "always cooperate" trajectories that ultimately degenerate the quality of the resulting policy by making it less likely to retaliate after a defection by the opponent.
the action-values in the opponent's
6.2 COIN GAME
Like (Zhao et al. 2022 we use GRU policy that has access to the current observation of the game plus both agents' actions in the previous turn train LOQA agent in the Coin Game. We run trajectories of length 50 with a discount factor y = 0.7 in parallel with a batch size of 8192. For evaluation run 10 seeds of fully trained agents for 50 episodes in league that involves other

---

Published as a conference paper at ICLR 2024
Random
LOGA
POLA
MFOS
Return
LODA
0.30
0.17
0.16
0.35
-0.05
0.26
0.17
0.19
0.01
0.29
0.07
0.13
0.24
0.19
0.19
0.38
-0.03
0.24
0.16
0.19
0.01
0.03
0.41
0..2
MF05
0.1
0.06
0.01
0.01
0.32
0.42
0.20
0.0
0.35
0.38
0.32
0.35
0.63
0.11
AC
-0.1
0.32
0.29
0.03
0.35
00.77
0.05
-0.2
-0.00
-0.3
-0.4
D.26
0.24
0.20
0.05
0.53
0.01
Randem
0.15
0013
0.21
0.11
00.3
0.01
Figure 2: Average rewards after evaluating 10 fully trained LOQA, POLA, and M-FOS seeds against different agents in 3x3 sized Coin Game lasting 50 episodes. AC for always Cooperate, AD for always defect. Notice that fully cooperative agent achieves average reward of 0.35 against itself. LOQA is able to generate a policy that demonstrates reciprocity- cooperation.
agents and plot the time-avee tineraveraged reward. The results shown in Figure We also experimented with ablations of LOQA that either removed self-play during training or removed the replay buffer of past policy weights to train against. These ablations showed that these two elements, although not essential, improve the performance of LOQA agents in the Coin Game by making them more cooperative with themselves and less exploitable by always-defect agents. For details refer the Appendix section[[
In Figure we observe that LOQA agents are able cooperate with themselves as indicated by the high average reward of 0.3 which is close to the 0.35 of an always cooperate agent against itself. LOQQ agents are able to achieve high social welfare without being exploited by an always defect agent they achieve an average reward of -0.05, which is comparable to POLA's own -0.03. More importantly our agents are fully trained after only hours of compute time in an Nvidia A100 gpu, compared to the hours of training it takes POLA to achieve the results shown in Figure It should be noted that as compared to IPD, of actions, therefore agent must learn take coins before learning whether they should cooperate with their opponent or not. We also consider full histories as opposed to one-step. making the state space significantly larger.
Coin Game, cooperation and defection consist of sequences
6.3 SCALABILITY EXPERIMENTS
In this section, we test LOQA, POLA, and M-FOS on larger grid sizes in the Coin Game to evaluate their scalability. As grid size increases, the average distance between the agents and the coin also grows This added complexity challenges the learning of reciprocal cooperative behavior. For exam- ple, when the opponent takes the agent's coin, the agent must learn to take multiple steps retaliate. This behavior is less likely to be discovered by random actions on larger grids. Our experiments with different grid sizes illustrate LOQA's A's scalability properties compared POLA and M-FOS.
In assessing the performance of the agents, we consider two metrics: the achievement a predeter- mined performance threshold and the time taken to reach this threshold. For the latter, it is critical to acknowledge that the conceptualization of a 'step' differs between POLA and LOQA. The steps in POLA encompass numerous inner optimization steps, rendering direct comparison of performance

---

Published as a conference paper at ICLR 2024
Threshold
Normalized Return against Each Other
Normalized Return against Always Defect
Weak
20.05
2-1.2
Medium Strong
20.1
2-0.5
20.2
2-0.2
Table 2: Thresholds based on two main criteria of reciprocity- bastly--badddd- cooperative policy, The weak and medium thresholds are designed such that all agents pass them, while the strong threshold represents good performance.
Weak Threshold
Medium Threshold
Strong Threshold
15000
15000
30000
LOQA
LOQA
LOQA
Wall Clock Time
12000
M-FOS
POLA
12000
POLA M-FOS
24000
M-FOS
POLA
9000
9000
18000
6000
12000
3000
6000
0
4
6
3
5 6
4
6
7
Grid Size
Grid Size
Grid Size
Figure 3: Wall clock time vs grid size for three seeds of LOQA, POLA, and M-FOS on reaching different thresholds. Each data point indicates the first time its corresponding seed passed a certain threshold. The wall clock time is measured in seconds. Red triangles indicate LOQA's performance while blue circles visualize LOQA's performance. Dashed lines pass through the average time for runs that passed for their respective algorithm.
per step inconclusive. To facilitate an equitable comparison, we employ the wall clock time of both algorithms, executed under identical computational configurations (GPU, memory, etc.).
The thresholds are defined based on two principal criteria pertaining to reciprocity- tive policy. Firstly, we evaluate the agent's return against an "Always Defect" opponent; this serves to test the cessation of cooperation following lack of reciprocation from the opponent. Secondly, we consider the agents' return against each other, which serves as measure of cooperative behav- ior. This dual tual-threshold approach is pragmatic as it discerns the distinctive behaviors; an 'Always Defect' agent meets the first threshold but fails the second, whereas an 'Always Cooperate' agent satisfies the second but not the first. However, policy resembling 'Tit-for-Tat' satisfies both thresh- olds. Furthermore,
coopera-
the grid size grows the average returns of agents per step decrease since takes longer to reach the coin. Therefore, it is crucial to ensure our thresholds are consistent when evaluating for different grid sizes.
We normalize the returns to make our thresholds standard over all grid sizes. Specifically, we multi- ply the return by the maximum distance possible between the agent and the coin for given grid size. Since the grid is wrapped, the normalization value N is given by the Manhattan distance between the center of the grid and one of the corners. We call this the normalized return and calculate our thresholds based on it. We have three thresholds for our evaluation. The weak and medium thresh- olds are designed so that all the algorithms are able to reach them. However, as LOQA reaches much higher return on large grid sizes compared to POLA AA and M-FOS, we set the strong threshold to high performance. The specification of the thresholds values is shown Table 2
The results of our experiments larger grid sizes are illustrated in Figure 3 All LOQA runs meet the strong threshold for grid sizes up to 6, but at a grid size of 7, one run falls short of the strong threshold. In contrast, every POLA and M-FOS run fail to reach the strong threshold for grid sizes above 3. For further details on the training curves from these experiments, please see the appendix section D Figure also provides the evaluation metries for each algorithm, detailing their behaviors. Additionally, LOQA consistently achieves each threshold substantially faster, by at least one order of magnitude because of its lower time and memory complexity.
8

---

Published as a conference paper at ICLR 2024
Grid Size: Algorithm: LOQA
Grid Size: 7 Algorithm: LOQA
Always Defect
Each Other
0.50
-0.5
medum
-1.0
0.00
1000
2000
3000
4000
6000 6000
7000
1000
2000
3000 4000 5000 6000 7000
Grid Size: 7 Algorithm: POLA
Grid Size:
Algorithm: POLA
0.2
Always Defect
Each Other
-0.5
0.1
-1.5
5000 10000 15000 20000 25000 30000 35000
5000
10000
15000 20000 25000 30000 35000
Grid Size: 7 Algorithm: M-FOS
Grid Size: Algorithm: M-FOS
0.2
Always Defect
Each Other
-0.5
meddun
0.1
-1.0
0.0
2500
5000
7500
10000
12500 15000
2500
7500 10000 12500 15000 Wall Clock Time
Wall Clock Time
for 3 seeds of POLA, LOQA, and M-FOS the evaluation metrics for a 7x7 grid size: Normalized return vs. themselves (Self) and vs. always defect (AD). The wall clock time is measured in seconds. Note that the range of the x-axis different for each algorithm as POLA and M-FOS need more time.
Figure 4: Training
The complexity of LOQA is equivalent to the calculation of REINFORCE estimator, as is standard in RL. Unlike POLA, LOQA does not involve calculations for opponent optimization steps nor does it differentiate through a computational graph of said optimizations during each training step, giving LOQA greater computationoal efficiency compared to POLA. Additionally, the absence of a second- order gradient in LOQA reduces the variance of its gradient estimators.
The model opponent learning in POLA is restricted to a limited number of optimization steps. In scenarios with increased complexity and varying opponent policies, additional steps may be neces- sary to accurately represent the opponent's learning. This increase necessitates extended runtimes and increased memory allocation for storing the computational graphs required for differentiation, positioning LOQA as more efficient and economical in memory usage.
7 LIMITATIONS
LOQA is primarily limited by the assumption that the other player acts accordingly to an inner action-value function. As such, it unable to shape other opponents that not necessarily follow this assumption. In similar way. LOQA agents are designed for environments with discrete action spaces. Future work could explore relaxations that allow LOQA agents to shape other types of agents and learn in continuous action spaces.
CONCLUSION
In this paper we have introduced LOQA, a decentralized reinforcement learning algorithm that is able to learn reciprocity-basee cooperation in general sum environments at a lower computational cost than its predecessors. action-value function is controllable, and the assumption that their opponent's 's policy closely mir- rors it. As a result, LOQA agents are able to shape other LOQA agents by performing REINFORCE updates that can be efficiently computed in hindsight after collecting environment trajectories. This especially advantageous as demonstrated in the experimental setup, where LOQA agents confi- dently outperform POLA and M-FOS agents in terms of optimality and efficiency in the Coin Game. Therefore, LOQA stands out as a promising algorithm for tackling more complex and empirically- grounded social dilemmas.
do so, LOQA agents rely on the observation that their opponent's

---

Published as a conference paper at ICLR 2024
9 ACKNOWLEDGMENTS
The authors would like to thank Mila and Compute Canada for providing the computational re- sources used for this paper. We would like thank Olexa Bilaniuk for his invaluable technical support throughout the project. We acknowledge the financial support of Hitachi Ltd, Aaron's CI- FAR Canadian AI chair and Canada Research Chair in Learning Representations that Generalize Systematically. Special thanks to Shumichi Akatsuka for his insightful discussions. We would like to thank the JAX ecosystem Bradbury et al. (2018).
REFERENCES
Alekh Agarwal, Nan Jiang, Sham Kakade, and Wen Sun. Reinforcement Learning: Theory and Algorithms. 2021.
Maruan Al-Shedivar, Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments, 2018.
Robert Axelrod. Effective choice in the prisoner's dilemma. Journal of Conflict Resolution, 24(1):
3-25, 1980.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnes and Qiao Zhang. JAX: composable transformations Python+NumPy programs, 2018. URL http:
Tim Cooijmans, Milad Aghajohari, and Aaron Courville. Meta-value learning: general framework for learning with learning awareness, 2023.
Gregory Farquhar, Shimon Whiteson, and Jakob Foerster. Loaded dice: Trading off bias and vari- ance in any-order score function estimators for reinforcement learning, 2019.
Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric P. Xing, and Shimon Whiteson. Dice: The infinitely differentiable monte-carlo estimator, 2018a.
Jakob N. Foerster, Richard Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness, 2018h 8b.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018.
Dong-Ki Ki Kim, Miao Liu, Matthew Riemer, Chuangchuang Sun, Marwa Abdulhai, Golnaz Habibi, Sebastian Lopez-Cot, Gerald Tesauro, and Jonathan P. How. policy gradient algorithm for learning to learn in multiagent reinforcement learning, 2021.
Vijay R. Konda and John N. Tsitsiklis. Actor-critic algorithms, 2000.
Adam Lerer and Alexander Peysakhovich. Maintaining cooperation in complex social dilemmas using deep reinforcement learning, 2018.
Alistair Letcher, Jakob Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson. Stable opponent shaping differentiable games, 2021.
Chris Lu, Timon Willi, Christian Schroeder de Witt, and Jakob Foerster. Model-free opponent shaping, 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Lloyd Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(10) 1100, 1953.
10

---

Published as a conference paper at ICLR 2024
John von Neumann. On the theory of games of strategy. Mathematische Annalen, 100:295-320, 1928.
Timon Willi, Alistair Letcher, Johannes Treutlein, and Jakob Foerster. Cola: Consistent learning with opponent-leariigg awareness, 2022.
Ronald Williams. Simple statistical gradient-following following algorithms for connectionist reinforcement learning. Machine Learning. 8:229-256, 1992.
Stephen Zhao, Chris Lu, Roger Baker Grosse, and Jakob Nicolaus Foerster. Proximal learning with opponent-lcaringg awareness, 2022.

---

Published as a conference paper at ICLR 2024
APPENDIX
A EXPERIMENTAL DETAILS
A.1 IPD
Game specification We use finite prisoner dilemma of game length 50. The hyperparameters of training are indicated in the Table 3
Hyperparameter
Value
Actor Model
logits
Advantage Estimation Method
TD0
Gradient Clipping Mode
disabled
Entropy Regularization Beta
0
Actor Learning Rate
0.001
Q-value Learning Rate
0.01
Target EMA Gamma
0.99
Optimizer
adam
Batch Size
2048
Game Length
50
Epsilon Greedy
0.2
Reward Discount
0.96
Agent Replay Buffer Mode
disabled
Differentiable Opponent Method
n step
Differentiable Opponent N Step
2
Table 3: List of hyperparameters used in the training run.
Agent's architecture The agent's policy is modeled via 5 logits. We take sigmoid over these logits to indicate the probability of cooperation on each of Start, CC, CD, DC, DD states. As the critic, we GRU unit described in A.2
Advantage Estimation In our experiments we used TD-0 for advantage estimation. We estimate the value for state as (s²) Ea, Q(2,, at (ar Consequently, use тi + (5(++))) Vsr) as the advantage.
A.2 COIN GAME
GPU and Running Time: We run our Coin Game experiments for 15 minutes on single A100 GPU with 80 Gigabytes of GPU memory and 20 Gigabytes of CPU memory. We choose the last state of the training as our checkpoint which the state at the 6000 iteration.
Agent's 's architecture We use the same GRU module taken from POLA's codebase. It is a GRU with two dense layers before the GRU unit and single linear layer after GRU's output. We use one for actor and one for our critic.
Replay Buffer We maintain a replay buffer of agents visited during the training. has a capacity of storing 10000 agents. We also push agent every 10 step. During trainig sample an agent from the replay buffer uniformly.
Hyperparameters In table 4 we show the hyperparameters used for training the agent.
A.3 SCALABILITY EXPERIMENTS
GPU and Running Time: In our scalability experiments we train the LOQA agents for a maximum of two hours single node that has one A100 GPU with 80 Gigabytes of GPU memory system with 20 Gigabytes of CPU memory. For POLA we use a longer runtime of 10 hours on the same node configuration it takes longer achieve reasonable performance.
Hyperparameters: For POLA and M-FOS we take the hyperparameters as suggested on their re- spetive github repository. We those same hyperparameters for all the grid sizes. For LOQA,
12

---

Published as a conference paper at ICLR 2024
Hyperparameter
Value
Grid Size
3
Game Length
50
Advantage Estimation Method
TD0
Gradient Clipping Method
Norm
Gradient Clipping Max Norm
1
Actor Optimizer
Adam
Actor Loss Entropy Beta
0.1
Actor Learning Rate
0.001
Actor Hidden Size
128
Actor Dense Layers Before GRU
2
Critic Estimation
Mean
Critic Optimizer
Adam
Critic Learning Rate
0.01
Critic Target Exponential Moving Average's Gamma
0.99
Critic Hidden Size
64
Critic Dense Layers Before GRU
2
Batch Size
512
Reward Discount
0.96
Agent Replay Buffer Mode
Enabled
Agent Replay Buffer Capacity
10000
Agent Replay Buffer Update Freq
10
Differentiable Opponent Method
Loaded-Dicc
Differentiable Opponent Discount
0.9
Table 4: List of hyperparameters used in the training run.
the same hyperparameters as found work suitably on the 3x3 board. Although, increased the batch size of LOQA to be 2048 to ensure reduced variance, In our 3x3 experiments for running the league we reduced the batch size to 512 so we can run ten seeds faster.
B IMPLEMENTATION DETAILS
Using naive REINFORCE estimators compute the gradient with respect the differentiable action-value estimate of the opponent leads to a suboptimal performance of LOQA due to high variance. We instead use loaded-DiCE (Farquhar et al., 2019) implemenation of the actor loss. The implementation algorithm is provided in
C ABLATIONS
C.1 SELF-PLAY ABLATIONS
We achieve the results presented in the main paper by training one agent against itself Figure shows the performance of L-S (LOQA without Self Play) agents. The performance of L-S is close to the (LOQA with Self Play) although, L is stronger in cooperating with itself.
C.2 AGENT REPLAY BUFFER ABLATIONS
Inspired by (Mnih et al. 2013), we use a replay buffer with capacity to store 10000 agents. Every 10 training steps, we push copy of the agent's parameters to the replay buffer. Figure Sssoww the performance of L-R (LOQA without Replay Buffer). The replay buffer gets updated slowly with new agents, thus, it decreases the variance of the training process.
Note when we the replay buffer of agents, we train the agent against sampled agent from the replay buffer. The replay buffer filled with the past versions of the agent encountered in training.
13

---

Published as a conference paper at ICLR 2024
Algorithm L.QQA ACTOR LOSS with loaded-DiCE
Input: Trajectory of length T, actor policy n², opponent action- action-value function Q discount A
factor 7, action discount factor X, advantages La +- 0
for t = 2, 2,,,...... , do
w f- 0
(5 b)) t- Σk= - (sks ak. bk)
for ...... ,T do
w t- X w log (T (a, (s:)
0 X w log (N (at bt))
(st, b.) + (st, b,) (f(())) f(v))
end for Compute म using Q (st,
((rr,b,, b.) and (st, bz) according to equation La + La + A [log n'' (a)lss)) + log t (b/ls()
end for
return: LQ
function f(x)
return: exp(x- STOP.GRADIENT(2))
end function
L-R
AC
AD
Q
& Return
0.30
0.21
0.29
0.24
0.32
0.12
-0.15
L
0.5
0.30
0.12
0.27
0.17
0.35
-0.05
0.26
0.12
0.14
0.12
0.14
0.08
0.17
-0.28
0.4
L-R
0.21
0.14
0.16
0.07
0.46
-0.08
0.35
0.3
0.27
0.16
0.25
0.20
0.32
0.07
-0.18
0.2
L-S
0.29
0.12
0.25
0.15
0.35
-0.04
0.27
0.1
0.17
0.07
0.15
0.19
0.29
0.07
-0.13
P
0.0
0.24
0.14
0.20
0.19
0.38
-0.03
0.24
-0.1
0.35
0.46
0.35
0.38
0.35
0.63
0.11
AC
0.32
0.08
0.32
0.29
0.35
-0.27
0.05
-02
-0.05
-0.08
-0.04
-0.03
-0.27
-0.00
-0.53
-0.3
AD
0.12
0.17
0.07
0.07
0.63
0.00
0.53
-0.4
0.26
0.35
0.27
0.24
0.05
0.53
0.01
-0.5
R
-0.15
-0.28
-0.18
-0.13
0.11
-0.53
-0.01
0.6
Figure 5: Average rewards after evaluating 10 fully trained LOQA and POLA seeds against different agents in a 3x3 sized Coin Game. We abbreviate L:LOQA, L-R:LOQA without replay buffer, L- S:for LOQA without self-play, P:POLA, AC:Always Cooperate, AD:Always defect and RaRandom.
14

---

Published as a conference paper at ICLR 2024
D SCALABILITY EXPERIMENTS
LOQA'S ACTOR CRITIC UPDATE DERIVATION
In LOQA assume that the trajectories are generated by policies x² and #2, which induces different probability measure over trajectories
Pr (T) u( (aol 3 (bol so) P(s (50, do, bo)...
Our expected return must now be computed using this probability measure
v (p):= Twpp
R (T)] =E Pu in2
Σ
xtr yr (se, (st,
Therefore we can write
Voi (x) Vo Σ R (T) (T)
Σ (T) a Vr) (P (T))
(F) - oo
II 1=0
π (aels)) (
Σ Re (T)
Ve (log n (aclse) + log a² (belst))
00
R (2
Voi (log x (arls)) log #2 (bel(s)))
P
T
=
E
,,2 PrP
A A ar, b, Vo (log x (a((s))) + log म² =b(eess))
1=0
In reality, we use trajectories sampled from T² and n², to compute our own estimate of the actor- critic update so we have
T A (St, ar, Vo (log x x²((aas)) (sc) + log X² =b((e(ss))
V VorV = E PMPTA
Σ 1m0
t.n2
F REPRODUCIBILITY
For reproducing results the IPD and the Coin Game please visit this link This is an anonymized repository and the instructions for reproducing the results and the seeds are provided. The seeds we used are 42 51.
15

---

Published as a conference paper at ICLR 2024
LODA
POLA
M-FOS
02
02
Gra Stoo:
J0ng Deead
Eanh
0.6
0.6
0.4
Crd Sbb:
0.0
0.0
Maryy Dete
0.8
0.8
0.6
0.5
ard Siba:
0.0
0.0
-0.6
0.5
1.0
1.0
-1.6
115
0.8
0.8
0.2
9-2
Gld S24-
0.0
0.0
.922
A0093 Deed
0.5
0.5
-1.0
-10
-1.5
30000 -1.5
Wall Clock Time
Wall Clock Time
Wall Clock Time
Figure 6: Training curves for 3 seeds of POLA and LOQA for 3x3, 4x4, 5x5 and 6x6 grid sizes the two metrics in the Coin Game: Normalized return vs. each other and vs. always defect (AD). The wall clock time is measured in seconds. Note that the range of the x-axis is different for POLA, LOQA, and M-FOS as POLA and M-FOS take longer to pass the considered thresholds.
16


Reviews:
[{"review_id": "8eiUCb3j53", "reviewer": "Reviewer_F4Df", "summary": "In this paper, the authors propose a decentralized multi-agent learning algorithm that fosters cooperation among agents even in adversarial settings, which they term as partially competitive environments. They provide experimental validation with Iterated Prisoners' Dilemma and Coin Game. Their key claim is that their proposed algorithm achieves state-of-the-art performance in these games with low computational cost. The authors here assume that each agent has access to the Q values of all other agents or can estimate them using the observations and rewards of all other agents.", "questions": "1. Can the authors concretely define \"partially competitive settings\"?\n2. What is the information structure assumed for each agent? If each agent can see the entire world state and also observe the actions of the other agent, then the policy for each agent should also depend on the history of the actions of the agents in addition to the current state for no loss of optimality.\n3. How can the opponent's true Q function be replaces with an estimate by the agent? Are we assuming that opponent observations and rewards are common information in the game? This is often not the case in real-worls multi-agent settings.\n4. It is not clear if LOQA will extend to all general-sum games. Also, the sub-optimality for each agent due to the modified objective function needs to be quantified and bounded. Are the authors proposing LOQA only for sequential social dilemmas?\n5. Can the authors establish or reason about the solution concept achieved using LOQA? Will it be a socially optimal solution. In non-symmetric games what will be the extent of sub-optimality for each agent?\n6. By modifying the objective function (returns) optimized by each agent, LOQA changes the underlying game. Can the authors show that a Nash equilibrium (or any refinement of it) of this modified game is a socially optimal solution of the original game?\n7. Also, will decentralized REINFORCE algorithm lead to attaining a Nash equilibrium in the above-defined modified game?\n8. How does this analysis extend to the case when all agents in a game are using LOQA?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors try to address the computational challenges faced by other MARL algorithms for sequential social dilemmas, by proposing an algorithm where each agent maintains an estimate of the Q values of all its opponents in order to determine its own policy improvement.", "weaknesses": "1. There are several papers in literature that provide decentralized algorithms to achieve individually and socially optimal solution in sequential social dilemmas. One of the criticism of these papers is the additional information needed by these algorithms, which is often not available in the real-world. This paper also has the same limitations. \n2. I think the novelty in the proposed method is limited based on the papers cited by it. The key idea is that each agent model the opponents policy using the rewards obtained by the opponents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8HCDRu7qhO", "reviewer": "Reviewer_xNg3", "summary": "This paper introduces learning with opponent Q-learning awareness (LOQA) which optimizes cooperation in mixed-motive environments by assuming the opponent samples actions proportionally to Q values. This method is computationally lighter compared with prior works.", "questions": "1. (Comments) The authors should use \\citep{} for (Author, Year) citations\n2. Any intuitions on why the other work[1], meta-game + model-free opponent-shaping does not work on coin game?\n\n[1] Lu, C., Willi, T., De Witt, C. A. S., & Foerster, J. (2022, June). Model-free opponent shaping. In International Conference on Machine Learning (pp. 14398-14411). PMLR.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. I like the idea of deriving some cooperative solutions in mixed-motive games without computing the meta-game solutions. There were many efforts in this direction but only a few paid off, the main limitation lies in the scalability of the multi-agent problems.\n2. The paper is clear and easy to follow.", "weaknesses": "In general, the experimental part has room for improvement\n1. When this line of research on LOLA has a few prior works, a comparison with a decent amount of previous works is necessary so that we know the proposed method is better. The good performance of a particular method under a particular environment is not the reason to abandon other methods, especially when POLA [1] did not compare with M-FOS [2]\n2. Results on the IPD and coin environment may be a bit preliminary when we jointly consider the contribution of the algorithm (efficiency). More complex games like Meltingpot 2.0 [3] with some clearly diverse background agent policies (cooperating for different variations of time and defect afterward) can be a good benchmark for the completeness of the experiments\n\n[1] Zhao, S., Lu, C., Grosse, R. B., & Foerster, J. (2022). Proximal Learning With Opponent-Learning Awareness. Advances in Neural Information Processing Systems, 35, 26324-26336.\n\n[2] Lu, C., Willi, T., De Witt, C. A. S., & Foerster, J. (2022, June). Model-free opponent shaping. In International Conference on Machine Learning (pp. 14398-14411). PMLR.\n\n[3] Agapiou, J. P., Vezhnevets, A. S., Duéñez-Guzmán, E. A., Matyas, J., Mao, Y., Sunehag, P., ... & Leibo, J. Z. (2022). Melting Pot 2.0. arXiv preprint arXiv:2211.13746.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TgmKeVXaqA", "reviewer": "Reviewer_rBSb", "summary": "The paper proposes LOQA -- a new member of the LOLA family (next to COLA and POLA). Conceptually, LOQA is different in that it assumes that the opponent uses Q-leaning (more specifically, that the probability of its action in a state is proportional to the corresponding Q-value relative to Q-values of other actions). The advantage of this algorithm is that it can achieve performance comparable to POLA (previous state-of-the-art of the family) but faster, which also improves scalability to bigger environments. This claim is complemented by experiments in the Coin Game, which is a two-agent coin collection game on a small grid with 2 agents.", "questions": "- I am not sure why in 5.1 the Q-value $\\hat{Q}^2$ is approximated as an empirical return. Since we need access to the real $Q^2$ or its approximation through opponent modeling regardless, can we not use a 1-step or n-step temporal difference instead (which would give a better bias-variance)?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Solutions to conditional (equilibrium-based) cooperation in general and improvements of LOLA in particular are relevant to MARL.\n- The paper is straightforward and well-written.\n- The experiments are sound, I especially like fig. 2.", "weaknesses": "- Related work lacks discussion of MARL approaches to learning prosocial equilibria other than reciprocity-based or opponent-shaping-based, such as reward redistribution https://ala2020.vub.ac.be/papers/ALA2020_paper_45.pdf https://arxiv.org/abs/2004.13332, mediation https://arxiv.org/pdf/2306.08419.pdf, contracts https://arxiv.org/pdf/2208.10469.pdf, and similarity-based equilibria https://arxiv.org/pdf/2211.14468.pdf.\n- Some limitations of previous LOLA-based approaches that LOQA does not fix are unmentioned in the Limitations section: e.g., it is only applicable to environments with 2 agents.\n- The contribution is limited to modifying an existing algorithm and improving its speed, but not performance. The new algorithm is also quite complex. I do not think the contribution is sufficient for ICLR. A workshop would be a better fit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6NKd6pl6z1", "reviewer": "Reviewer_SYs7", "summary": "This paper presents a method for decentralized learning in general-sum games, building on the assumption that adversaries sample actions according to their action-value function. The experiments demonstrate significant improvements in wall-clock time and performance against POLA in the coin game.\n\nI am recommending borderline acceptance for this work, as the method is a useful contribution towards scaling LOLA-based methods. However, I would like to see the evaluation strengthened with more and increasingly complex environments, as well as more than one baseline (particularly M-FOS).", "questions": "1. Nitpick: The scaling of LOQA in figure 3 would be clearer with a log-scale y-axis.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The related work is detailed, covering LOLA and meta-learning approaches to social dilemmas.\n2. The method is clearly motivated and described. The underlying assumption is well explained and plausible.\n3. The evaluation demonstrates a significant improvement in computational efficiency against POLA, with the analysis demonstrated in figures 3 and 4 showing this scaling behavior well.\n4. Figure 1 presents a validation that LOQA is capable of learning tit-for-tat-like strategies in IPD.", "weaknesses": "1. The evaluation is limited in diversity, comparing LOQA to a single baseline algorithm (POLA) on a single environment (coin game). The selection of POLA vs alternative LOLA extensions is justified, but I am unsure how M-FOS is neglected when it has also been demonstrated to be effective at the coin game? Furthermore, the choice to only evaluate against POLA on the coin game is limiting, when full-history IPD and chicken would be equally interesting and strengthen the results.\n2. The scaling performance of LOQA would be better demonstrated by including further, more complex environments, rather than just scaling up the grid size of the coin game. Since previous LOLA extensions have been prohibitively expensive, this has not been possible, but it appears LOQA runs in a very reasonable amount of time on the largest of these tasks. This gives the opportunity to demonstrate LOQA on a \"more complex and realistic scenario\", rather than just extrapolating from its scaling performance here.\n3. Whilst the training dynamics of POLA are consistent across seeds, LOQA seems to have significant variance (Figure 4). The number of LOQA seeds should be increased beyond 3 to handle this variance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uEMyzmtlb4", "reviewer": "Reviewer_iBDc", "summary": "the paper presents LOQA, a decentralized reinforcement learning algorithm that optimizes individual utility while promoting cooperation among adversaries in partially competitive environments. LOQA is designed to achieve state-of-the-art performance in benchmark scenarios like the Iterated Prisoner's Dilemma and the Coin Game, making it a promising approach for practical multi-agent applications. The paper provides a detailed description of the LOQA algorithm, including its opponent Q-learning awareness assumption, and presents experimental results that demonstrate its effectiveness.", "questions": "Q1: The IPD and coingame mentioned in the paper mainly refer to two general-sum games. Can the algorithm be applied to other types of environments, such as zero-sum games, or more complex gaming environments, such as StarCraft II?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Quality: The paper provides a detailed description of the LOQA algorithm, including its opponent Q-learning awareness assumption, and presents experimental results that demonstrate its effectiveness. The experiments are well-designed and the results are statistically significant.", "weaknesses": "the LOQA algorithm assumes the opponent acts accordingly to an inner action-value function and is designed for environments with discrete action spaces. This means that LOQA is unable to shape other opponents that do not necessarily follow this assumption. The assumptions and dependencies of algorithms are strong, making it difficult to handle continuous action space problems and multi-agent issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)