Context and History Aware Other-Shaping
Authors: Akbir Khan
Newton Kwan
Timon Willi
Chris Lu
Andrea Tacchetti
Jakob Nicolaus Foerster
Venue: Submitted to ICLR 2023 (2023)
Keywords: Shaping
Multi-Agent
Reinforcement Learning
Meta Reinforcement Learning

Abstract: Cooperation failures, in which self-interested agents converge to collectively worst-case outcomes, are a common failure mode of Multi-Agent Reinforcement Learning (MARL) methods. Methods such as Model-Free Opponent Shaping (MFOS) and The Good Shepherd address this issue by shaping their co-player’s learning into mutual cooperation. However, these methods fail to capture important co-player learning dynamics or do not scale to co-players parameterised by deep neural networks. To address these issues, we propose Context and History Aware Other-Shaping (CHAOS). A CHAOS agent is a meta-learner parameterised by a recurrent neural network that learns to shape its co-player over multiple trials. CHAOS considers both the context (inter-episode information), and history (intra-episode information) to shape co-players successfully. CHAOS also successfully scales to shaping co-players parameterised by deep neural networks. In a set of experiments, we show that CHAOS achieves state-of-the-art shaping in matrix games. We provide extensive ablations, motivating the importance of both context and history. CHAOS also successfully shapes on a complex grid-worldbased game, demonstrating CHAOS’s scalability empirically. Finally, we provide empirical evidence that, counterintuitively, the widely-used Coin Game environment does not require history to learn shaping because states are often indicative of past actions. This suggests that the Coin Game is, in contrast to common understanding, unsuitable for investigating shaping in high-dimensional, multi-step environments.

Under review as a conference paper at ICLR 2023
CONTEXT AND HISTORY AWARE OTHER-SHAPING
Anonymous authors Paper under double-blind review
ABSTRACT
Cooperation failures, in which self-interested agents converge to collectively worst-case outcomes, are a common failure mode of Multi-Agent Reinforce- ment Learning methods. Methods such as Model-Free Opponent Shaping and The Good Shepherd address this issue by shaping their co-player's learning into mutual cooperation. However, these methods fail to capture important co-player learning dynamics or do not scale well to co-players parameterised by deep neural networks. To address these issues, we propose Contexi and History Aware Other- Shaping (CHAOS). A CHAOS agent a meta-learner that learns to shape its co- player over multiple trials. CHAOS considers both the context (inter-episode in- formation), and history (intra-cpisode information) to shape co-players. CHAOS also successfully scales to shaping co-players parameterised by deep neural net- works. In a set of experiments, we show that CHAOS achieves state-of-the-art shaping in matrix games. We provide extensive ablations, motivating the impor- tance of both context and history. CHAOS also successfully shapes on grid- world-based game, demonstrating CHAOS' scalability empirically. Finally, we provide empirical evidence that, counterintuitively. the widely-used Coin Game enviromment does not require history to learn shaping because states are often indicative of past actions, making it unsuitable for investigating shaping.
INTRODUCTION
Multi-agent learning has shown great success in strictly competitive (Silver et al. 2016) and fully cooperative settings (Foerster et al., 2019 Rashid al. 2018). In competitive games, agents can learn Nash equilibrium strategies by iteratively best-responding to suitable mixtures of past oppo- nents. Similarly, best-responding to rational co-players leads to the desirable equilibria in coopera- tive games (assuming joint training). In contrast, Nash equilibria often coincide with globally worst welfare outcomes in general-sumt games, rendering the aforementioned learning paradigms ineffec- tive. For example, in the iterated prisoner's is dilemma (IPD) (Axelrod & Hamilton, 1981 Harper ct al. 2017), naive best-response dynamics converge on unconditional mutual defection Foorster al. 2018).
The above methods ignore a crucial factor: when multiple learning agents interact in a shared envi- ronment, the actions of one agent influence the environment and, often, the reward of other agents, which in turn influences their learning dynamics. For example, a car merging into the middle lane in heavy traffic makes it unattractive for fellow collision-averse motorists to move to the middle lane at the same time. Our paper investigates methods which allow agents to exploit this interconnection between their actions and the learning outcome of other agents and leverage it their advantage. Such "shaping" methods explicitly account for other agent's s learning and have achieved promising results, e.g. discovering the prosocial tit-for-te tat strategy in the IPD (Foerster et al. 2018 Letcher et al., 20196] Willi et al. (2022) Balaguer et al. 2022 Lu et al. 2022). However, early shaping meth- ods are myopic (only shape the next learning step the co- co-player), require white-box access the co-player's parameters and require higher-order derivatives. To overcome these shortcomings. both Model-Free Opponent Shaping (Lu et al. 2022 M-FOS) and The Good Shepherd (Balaguer et al.) 2022 GS) frame shaping as meta reinforcement learning problem. In these approaches, the meta- agent learns to shape others by observing full training runs of the co-players in each meta-training episode before updating its policy. M-FOS and GS showed promising empirical success. However, both methods have shortcomings: M-FOS's meta-agent outputs a policy parameterisation for the inner-agent (similar to HyperNetworks (Ha et al. 2017)). This limits M-FOS to games where the

---

Under review as a conference paper at ICLR 2023
policies can be represented compactly, such as in infinitely-iterater matrix games. While M-FOS reports results in a higher-dimensional game (in which neural networks represent the policies), uses hierarchical architecture. GS does not output whole parameterisations but instead keeps its policy fixed during the entire duration of a trial. This prevents GS from using the training context to shape the co-player adaptively.
To address both issues, we propose Context and History Aware Other-Shaping (CHAOS). In CHAOS, the meta-agent and the inner agent it controls are parameterised by a single recurrent neural network (RNN). A CHAOS agent meta-learns by retaining its hidden state throughout an en- tire meta-episode, similar RL (Duan et al., 2016) in single-agent RL. This hidden state enables CHAOS agents to react to two components of the co-player's learning: The context inter-episode learning and the history- intra-episode behaviour. In shaping problems, history captures the co- player's current policies whilst context captures the co-player's learning rules. Together these two enable CHAOS to dynamically shape agents. Combining the meta-agent and the inner agent into one recurrent meta-learner avoids outputting policy parameterisations, unlike M-FOS.
We show that CHAOS discovers a ZD-extortion-like strategy in the finitely-itrrtted prisoner's dilemma (a more challenging setting than the infimitely-iterttedd PD where the environment is non-differentiable and where policies cannot be represented compactly). Moreover, we show that CHAOS matches or outperforms GS and M-FOS in iterated matrix games. CHAOS also matches state-of-the-ar shaping against memory-based agents in the Coin Game, a grid-based environment where deep neural networks represent policies.
To summarise our contributions
We introduce CHAOS, a shaping method capturing both learning context and history, suit- able for high-dimensional games.
We formalise the concept of history and context for shaping and analyse their respective roles empirically.
We demonstrate state-of of-the-ar art performance set of iterated matrix games.
We identify fundamental problem in the widely-used Coin Game
2 RELATED WORK
Opponent Shaping Many methods exist that explicitly account for their opponent's 's learning. Just like CHAOS, these approaches recognise that the actions of any one agent influence their co-players policy and seek to use this mechanism to their advantage (Foerster et al. 2018 Letcher et al. 2019a] Kim et al.) 2021a Willi et al. 2022). However, in contrast to CHAOS, these approaches require privileged information to shape their opponents. These models also myopic since anticipating many steps is intractable. BBllggeer et al. (2022) and Lu et al. (2022) solve the issues above by framing opponent shaping as meta reinforcement learning problem, which CHAOS inherits and
builds upon. The specific differences to M-FOS and GS will be the subject of Section 4
Opponent Modelling Similarly to work, opponent modelling tries to disentangle some aspects of other agents' policies from the environment (Mealing & Shapiro 2017 Raileanu et al. 2018 Tacchetti et al. 2018). In contrast to our work, these approaches do not consider agents learners.
Furthermore, they do not observe agents at different stages of learning and thus whilst modelling as non-stationary, do not observe learning dynamics (Synnaeve Bessière 201T). Finally, CHAOS does not explicitly model any aspect of the opponent.
Multi-Agent Meta-Learning Multi-agent meta-learning approaches have also shown success in mixed-game with other learners (Al-Shedivat et al.] 2018 Kim et al. 2021b: Wu et al., 2021)
Similar to CHAOS, they take inspiration from meta reinforcement learning their approach to learn the optimal initial parameterisation for the shaper akin to Model-Agnostic Meta Learning FFin et al. 2017). h. In contrast, CHAOS uses an approach similar to RI." (Duan et al. 2016). which trains an RNN-based agent to implement efficient learning for its next task. Furthermore, CHAOS is optimised using evolution strategies (Salimans et al. 2017), allowing it to consider much longer time horizons than policy-gradicss metods (Schulman al. 2017).
"Other" breaks with the line of seminal work on opponent shoping, but highlights the general-sum aspect
2

---

Under review as a conference paper at ICLR 2023
Opgonunt
Ab
Oo,
(1;
P1,
$1.
Ph.
Ps
Feionde
Fyrinnle
Episode
Episode
9p, ho
0p.
0g.
Og.
0g.
9g hg
011
CHAOS
Figure I: Example meta-learning interaction. CHAOS / (blue) resets its hidden state h at the begin- ning of trial, updates its hidden state after each environment interaction, then updates its parameters at the end of trial. The opponent é (orange) resets its parameters at the beginning of trial, then updates its parameters at end of each episode. Optionally, if the opponent uses memory, its hidden state h is reset at the beginning of trial and updated after each environment interaction.
BACKGROUND
Partially Observable Stochastic Game (POSG) A POSG is given by the tuple M (W,A, O, S, T,I,, R, 7). where A, O, and S denote the action, observation, and state space, respec- tively. These parameters be distinct at every time step and also incorporated into the transition function x A AS, where = A the joint action of all agents. Each agent draws individual observations according the observation function I S N + O and obtains a reward according to their reward function R S x A x N + R where N = [1.....n.. POSGs can represent general-sum games. The single player case, {1}, of POSGs are Partially Observable Markov Decision Processes (POMDPs).
Evolution Strategies (ES) CHAOS uses Evolution Strategies (Salimans 2017 ES) optimise the meta-agent. ES is model-free optimisation method. Let F Iaa -y R be some function we want to optimise over. Instead of optimising the objective directly, ES blurs F(x) with Gaussian noise
(e [F ae)]
where a is following simple gradient estimator:
controlling how much Gaussian noise added. This allows using the
Vx VxEeeNN(o.L,)FFs + aε)] E(- [ F +
allowing the optimization of non-differentiable functions using gradient descent methods. ES by- passes the credit assignment problem by directly optimizing in the parameter space of the model instead of the policy space and is thus suitable for long-time horizon problems (Salimans et al. 2017].
RL3 CHAOS takes an RLL-like approach meta-learning. RL2 is a single-agent meta- reinforcement learning method where the meta-agent is parameterised by recurrent neural network (ò, f), where @ are the network parameters and i the hidden state. RL samples MDPs from distribution Pp P + R+ and interacts with each MDP sequentially over a trial, consisting of number of episodes trial and resets only when it faces new trial, i.e. new MDP is sampled. The objective is to maximise the expected total discounted reward over a trial instead single episode
Importantly, the RL² agent retains its hidden state across all episodes in
L
ESst
Σ 1'r (st, ar)
where - K E and K is the episode length. The RL² agent is thus encouraged to use all the information captured in its hidden state RL agents have been shown to scale to high- figh--imensionall problem settings (Duan al. 2016).

---

Under review as a conference paper at ICLR 2023
Table Converged Reward (row, column) for agents against Naive Learners on the Iterated Prison- ers Dilemma (IPD), Iterated Matching Pennies (IMP) and Coin Game, reported across 5 seeds with standard deviations. Reward is averaged per (step/episode) for (Matrix/Coin) game respectively.
IPD
IMP
Coin Game
CHAOS
-0.13 + 0.02, -2.84 + 0.05
0.86 - 0.02, -0.86 + 0.02
6.51 = 0.46, 2.71 = 0.11
M-FOS
-0.60 + 0.14, -2.34 + 0.14
0.83 0.09, -0.83
0.09
2.67 + 0.52, 3.94 + 0.15
GS
-0.97 + 0.03,-1.22 + 0.10
-0.01 + 0.01, 0.01 + 0.01
6.72 + 0.72, 2.39 + 0.10
NL
-2.00 + 0.00, -2.00 + 0,00
0.00 + 0.00, 0.00 + 0.00
0.47 + 0.83, 0.26 + 0.30
0.7
Evaluation over 1 Trial
10
Probability of Cosperation
Stale Visitation Frequency
State Visitation
GHAGS
APO
PiGooperation)
6.8
0.
DC
Reward
-1.2
DD
DD
-1.5
START
START
6.4
-2.0
-25
c22
0.-
-30
40
0.0
0.0
20
Episode
Episode
Episode
(a)
(b)
(c)
Figure 2: Evaluation results in the finite IPD over a single trial composed of 100 inner episodes over 20 seeds, where (a) shows the reward, (b) CHAOS' conditional probability of cooperation, and (c) state visitation.
Good Shepherd GS formalises shaping as a meta-learning problem over sequence of trials of length T. Each trial contains E episodes. In each trial, GS shapes a new co-player in a POSG M, where (Φ., o_;) correspond to GS' and the co-players' parameters, respectively. During trial t, GS uses a fixed policy oi. At the end of each inner episode, the co-players update their parameters with respect to the episodic return Jh -E (0f; &L;), where K is the length of an episode. For example, if the co-players were Naive Learners, the update looks as follows: GS optimises the meta-return >. Jf (summed over all episodes) at the end of a trial using Evolution Strategies (Salimans al. 2017). The policy is parameterised by feed-forward network, thus lacking both history and context. Therefore, GS cannot adapt to changing learning dynamics of the co-player, which is detrimental, as we show in Section
Model-Free Opponent Shaping M-FOS frames opponent shaping as a meta reinforcement task. More specifically, the meta-task is formulated as POMDP (S, A, O, R, 7) over an under- lying general-sum game, represented by POSG M. In the POMDP, the meta-state S spans the policies of every player in the underlying POSG: & (Φf ) E S. The meta-action space A consists of the policy parameterisation of the underlying inner agent playing the game for the meta-agent, At each meta-episode, conditioned on both agent's 's policies, M-FOS outputs parameters of the next inner-agents (similar HyperNetwork (Ha et al. 2017)) i.e., d. of To (- oe), where 0 is the parameters of the M-FÓS agent. The meta-reward is the return of the inner agent over one inner episode, F- o rf df:). This Hypernetwork-lik approach is M-FOS' main shortcoming is difficult to scale to complex inner-agent policy parameterisations. To scale to complex inner-agent policy parameterisations that are used in the Coin Game environment, they use a hierarchical architecture in which the meta-agent instead outputs conditioning vector for the inner agent that contains context information (Lu et al.) 2022). M-FOS is optimised using Evolution Strategies (Salimans et al. (017)) for the itcrated matrix games and uses PPO (Schulman et al. 2017) for the Coin Game environment.
4 METHOD
We assume a set of POSGs M and distribution we can sample from PM- We also assume a set of co-player initialisations of and a corresponding distribution Pa. as shaping acknowledges other learners within the environment, in contrast to RL2, Just like in GS, we define inner episode to be a finite sequence of interactions within fixed POSG M (N, A, O, S, T.I, R, and fixed initial learner, and a trial to be a sequence of inner episodes.
4

---

Under review as a conference paper at ICLR 2023
Figure illustrates the interaction between agents and the environment. First, the shap- ing agent's s parameters de and hidden state h, are initialised. At the start of a trial, co-players Φ-i Ps and new game (POSG) PM are sampled. During an episode of length K, upon receiving a state, agents take their respective ac- tions, o. (. (- of h)). At each time step in the episode, the internal state of the shaping f(o.. ht). On re- ceiving actions, the POSG returns rewards r/.
Algorithm 1: CHAOS update algorithm. Given a distribution of game functions M ~ PM, returns Jr, J-i, policies Td. * and their respective initial hidden states h4, h- and distribution of initial co-players pg, this algorithm updates a shaper policy o over T trials consisting of E episodes.
Require: 6; Ps. PMr E, T
agent is updated: h
1: for t 0 T do
2: Initialise trial reward J = 0
3: Initialise shaper hidden state h; =
new observations +f and done flag d, indi- cating if an episode has ended.
4: Sample co-players Φi P
5:
for e = 0 to E do
When an inner episode terminates, the co- player takes a gradient update maximising to- tal episode return, J Ek= (6), &=,).
6:
Sample game M PM
7:
Initialise co- co-players' h- =
8:
J-i, M₁. hi; = His
The updated co-player xل; and the shaper's hidden state h,K are passed to the next episode. This process is repeated over E episodes in trial. When a trial terminates, the shaper's pol- icy is updated, maximising total trial reward,
9:
Update co-players'
a_ Φ- + V Vo
10:
h
- h1
J J-
12:
end for
13: Update @e with respect to J
J= E. Jf. via ES. This leads the follow-
14:
end for
ing objective,
max max P(O). [J] 4
We use Evolution Strategies (Salimans et al. (017) to maximise the expected fitness across a pop- ulation. At the start of each generation, a population of CHAOS agents of size M and N Naive Learners (NL) are initialised. During a generation, the population plays against copies of the Naive Learner in parallel for a series of E inner episodes. At the end of inner episode, each copy of the Naive Learner performs a gradient update. At the end of the trial, CHAOS performs a gradient update in the direction of the maximum expected fitness, and the next generation begins.
Difference to M-FOS and GS CHAOS can be understood as a simpler version of M-FOS in which the meta-agent and underlying agent are collapsed into a single agent. To contrast CHAOS to M- FOS in more detail, we differentiate between the method definition and the actual architecture.
In the method definition of M-FOS, there are two action spaces: the meta-action space A and the underlying action space A. In CHAOS, the only action space is that of the underlying game, mean- ing there is only one agent in CHAOS, whereas there are two agents in M-FOS. Consequently, CHAOS can be understood as a special case of M-FOS. To make this explicit, let us define his- tory as trajectory of an episode e, Te (og, ag. r re..... r.K) and context as a trajectory of a trial t, F. = (To...., T.). Thus we can express the policies of CHAOS, M-FOS and GS as the following:
CHAOS:
a ~ To( ( Te,
M-FOS:
a ToNTo(+17)) Te) a a~
GS:
Thus becomes apparent that 1) M-FOS' use of an outer agent redundant and 2) that GS is unable to condition upon the context.
Secondly, CHAOS' architecture is different from the architectures proposed in M-FOS. M-FOS pro- poses two significantly different architectures (Lu et al. (202)). For matrix games, where table can represent the policy, the meta-agent a feedforward neural network whose output is the exact policy of the underlying agent. In this case, the meta-agent is trained using Evolutionary Strategies. For the Coin Game, in which a simple table cannot represent policies, M-FOS proposes an architecture akin to Hierarchical RL. Both the meta-agent and the underlying agent are recurrent neural networks. The underlying agent resets their hidden state after each episode, whereas the meta- agent does not. In this architecture, the meta-agent does not output the full parameterisation of the underlying agent
5

---

Under review as a conference paper at ICLR 2023
but instead outputs conditioning variable, which the underlying agent uses input. M-FOS relies on the assumption that outputting a conditioning variable is equivalent to outputting a policy pa- rameterisation. The conditioning variable is fixed during episode. In contrast, CHAOS only uses one recurrent neural network that does not reset its hidden state after an episode. Both M-FOS and CHAOS can capture context and history; however, CHAOS only needs one agent to do so.
Next, we compare CHAOS to GS. Originally, GS does not use a recurrent agent, and there is no discussion about notions of history or context. Thus GS cannot capture history context. Even though it is not discussed, the framework is extensible to a recurrent meta-agent. However, that meta-agent only captures history as the hidden state is reset after each episode. By not capturing context, GS fails to shape in zero-sum games, such as the Matching Pennies, which we show in our experiments.
EXPERIMENTS
5.1 ENVIRONMENTS
Iterated Prisoner's Dilemma is well-known and widely studied general-sum game illustrating that two rational agents may not cooperate even it is globally optimal. The players choose to either cooperate (C) or defect (D) and receive a payoff according to Table 2a In the iterated prisoner's dilemma (IPD), the agents repeatedly play the prisoner's dilemma and can observe the previous decisions.
Past research has used the infinitely IPD in their experiments (Foerster et al. 2018 Letcher et al,
2019b Willli al. [022 Lu et al. 2022 Balaguer et al. (022). In the infinite version, players submit a policy represented by five parameters, where each parameter is the probability of cooperation after each state given a one-step history (CC, CD, DC, DD, start). Press & Dyson (2012) showed that having access to the last state is sufficient for acting optimally. The infinite version is a differentiable game Balduzzi et al. 2018) as the exact value function can be calculated directly from the policies, thus accessing exact gradients is possible and optimization tractable. In our work, we consider the finitely IPD (f-IPD), and we do not take advantage of exact value functions, resulting of the game that is more similar to current reinforcement learning environments. In the f-IPD, the agents do not submit a full strategy but take an individual action (either C or D) at each timestep.
version
Over repeated interactions, IPD produces spectrum of interesting behaviours. In particular, two cases are of interest in this work: 1) Cooperation, and 2) Beero--Dteemmnnnt (ZD) Extortion strategies. In Cooperation, agents are shaped sufficiently to CC and choose not defect, even though this would increase short-term rewards. In ZD-Extortion (Press & Dyson 2012), co-players cooperate while allowing the shaper enforce linear relationship between their player, thus inducing behaviours that are more favourable than mutual cooperation.
payoff and that of the co-
Iterated Matching Pennies (IMP) is an iterated matrix game like the IPD. The players choose either heads (H) or tails (T), and receive a payoff according to the choices of both players. In contrast to the IPD, which is general-sum game, IMP is a zero-sum game. The only equilibrium strategy for each one-memory agent is to play a random policy, resulting in an expected joint payoff of (0,0). It only with intra-episode memory that shaper can observe a co-player's policy and begin shaping.
Coin Game is a agltt, wrap-around grid world environment that simulates social dilemmas (like the IPD) with high-dimensional states and multi-step actions (Lerer & Peysakhovich] 2017). Two players- - blue and orange- move around a grid and pick up blue and orange coloured coins. When player picks up coin of its own colour, the player receives a reward of +1. When a player picks up a coin of the co-player's colour, the player also receives a reward of +1 and the co-player receives reward of -2. If a coin gets picked up, new coin of the same colour appears in a random location on the grid. If both agents reach a coin simultaneously. then both agents pick up that coin (the coin is duplicated). The episodes are of fixed length. When both players pick up coins without regard to colour, the expected reward is O. In contrast to the IPD, the Coin Game requires learning from high-dimensional states, task that current shaping methods struggle learn.

---

Under review as a conference paper at ICLR 2023
Probability
State Visitation
10
State Visiation Frequency
0.75
0.8
6.8
Rewerdd
0.50
P(Heeds)
6.6
TH
0.00
START
START
-2.33
c2
0.2
-0.78
20
60
80
100
02
100
Episode
Epieode
Episode
(a)
(b)
(c)
Figure 3: Evaluation results single trial (with novel co- co-player) composed of 100 inner episodes over 20 seeds for the Iterated Matching Pennies. (a) Average Reward (per step) (b) CHAOS' proba- bility of cooperation conditioned state and (c) state visitation.
5.2 BASELINE COMPARISONS
We compare our method against three baselines: a Naive Learner, M-FOS and GS. A Naive Learner (NI.) does not account for the learning of the co-player. updates at the end of each inner-episode with learning rate a:
1t+1 = of a 0 R (o).
(2)
In the IPD, our NL is parameterised as a tabular policy trained using PPO (Schulman et al. 2017)
In the Coin Game the NL is parameterised by a recurrent neural network trained using PPO. The specific implementation details are provided in Appendix D
For M-FOS and GS, we optimise both methods using Evolution Strategies (Salimans et al. 2017 in line with the original implementations (Lu et al., [2022] Balaguer et al.) (0022). For M-FOS, we use the hierarchical architecture used in its Coin Game results since we are using neural networks for these environments instead of simple tabular policies. The implementation details for M-FOS and GS are provided in Appendix [ and Appendix [D]
In every game, CHAOS is parameterised as recurrent neural network and trained using Evolution Strategies (Salimans et al., 2017). We used the Jax library (Bradbury et al. 2018) with the Haiku framework (Hennigan al., 2020) to implement our neural networks. For the Evolution strate- gies, we relied on the Evosax library (Lange) (022). Our experiments were performed on NVIDIA A40 and VIOO Addditional implementation details and hyperparameters for each game are provided in Appendix D Furthermore, the whole codebase will released upon acceptance.
5.3 ABLATIONS
Figure 4: Illustration a special state of the Coin Game where memory-less agents can infer conventions. If both agents are on the same square, we can infer that an agent has broke convention.
To evaluate the importance of context and history we apply the following ablations.
The Hardstop Challenge during trial, after & episodes the co-player no longer takes learner updates. In the situation where the co-player no longer updates, optimal behaviour would be exploit this fixed policy (effectively stop shaping). We choose k = 2 to be less than the number of episodes required for shaper to reach to ZD- Extortion-like policies. This challenge tests if shapers' can 1) identify the sudden change in a player's learning dynamics 2) react and deploy a more suitable exploitative policy. We evaluate CHAOS against GS, to compare context and context-less shaping methods.
The Only-History Challenge we reset the hidden state of CHAOS between episodes, removing its ability to context to shape (CHAOS reset memory). In this challenge, shapers must infer co- co-player's current policy by only using the history. We evaluate the shaper within the IMP environ- ment, different episode lengths, limit the relative strength of history.
7

---

Under review as a conference paper at ICLR 2023
Eniluation over 1 Tiid
Piebunility Cooperation
State Viatation
Coln Pioked Up
1:2-
PDD MM
53
Esiodo
Episode
Ep 8ode
Ephoode
(a)
(b)
(c)
(d)
Figure 5: Evaluation results over single trial (with novel-coplayer) compromising over 5 seeds for the Coin Game (a) Reward, (b) the Shaper's 's frequency of picking up its own colour coin, (c) state visitation, and (d) the number of coins picked up per episode.
6 RESULTS
In this section we report and compare the results between CHAOS and our baselines on our en- vironments. We find that CHAOS shapes achieves state-of-the-art results the IPD. On the IMP game, CHAOS achieves state-of-the-art results, outperforming M-FOS, GS and other baselines. Next we demonstrate that CHAOS is scalable, as it achieves comparable shaping in the Coin Game Finally through series of ablations fective shaping. Similar to Meta-RL test protocol, for the evaluation we evaluate the performance of shapers against novel POSG M and co-player initialisation Φ
demonstrate the importance of context and history for ef-
Iterated Prisoner's Dilemma In the IPD, we inspect the converged reward for each shaping algo- rithm against PPO agent. Here, CHAOS shapes its co-player more aggressively than the base- lines (see Table D. achieving an average return of -0.13 13 per episode. All shaping baselines reach ZD-extortion-like policy. CHAOS switches policies during co-player's learning, switching from cooperation to exploitation (see Figure ??). In Figure 2 we display the state visitation over one trial. It shows that a fully trained CHAOS agent pursues tit-for-tat like-strategy to encourage co- operation within the first 5 episodes before pursing an excessively exploitative policy. At this point, the co-player is shaped, as is it unable to move another equilibrium.
Iterated Matching Pennies In the IMP, CHAOS exploits its opponent to achieve a score of (0.80.-00 -0.80) (see [Tb As expected, GS cannot shape the opponent, achieving score close to the Nash Equilibrium. Without having any context, it is not possible to shape the opponent because the opponent can also switch a random strategy to at least achieve a score of O. Thus dynamic- shaping. is required to find exploitative strategies. We find that M-FOS another dynamic shaping method is able to find exploitation too.
Coin Game CHAOS outperforms M-FOS in Table I providing evidence that CHAOS is scalable to more complex policies. CHAOS demonstrate shaping, as indicated by the payoffs of the shaper vs the co-player. To verify that CHAOS has shaped (as opposed to simply obstructed the co-players learning) we also inspect player's ss ability to pick up coins see Figure (5..
Cooperation in the Coin Game means that players only pick up coins of their colour. Thus evaluating % of cooperating actions is useful. We adapt our analysis from the IPD to the Coin Game We extend the four states from the IPD (CC, CD, DC, DD) also include the start convention (before a player picks up a coin). At the start of an episode, the state is SS as neither player has committed cooperative or defective behaviour. The state remains SS until a player picks up coin and breaks one of the other 7 states. For example, observing a low probability of cooperating when shaper has been exploited (CD), demonstrates that shapers punish co-players who break cooperation.
Figure 5b demonstrates how CHAOS uses context to effectively shape its co-players. Shapers have two axes on which to evaluate their co-players, 1) how competent they are and 2) how cooperative they are. We measure competency as how many coins player can pick up. If a co-player is not competent yet (say at the beginning of its training), we would expect a shaper to exploit this behaviour. In Figure 5e, the difference between cooperating in SC and SS highlights how CHAOS uses context to evaluate exploitability of its co-player. In SS, when both agents have not picked up coins, CHAOS probes for exploitability by not cooperating, whereas in SC, where the co-player has already demonstrated it is competent, CHAOS follows convention. Figure 5c demonstrates how CHAOS exploits its co-player. We expect efficient shapers spend the majority of their time either
8

---

Under review as a conference paper at ICLR 2023
Onlyviattoy Challinge, IMP (L-2)
Orky-History Challenge, IMP (.-2)
Onty-Hsttryy Challlnge, MP (.=100)
1.00-
1.30
100-
0.70-
Unios
0.75
CHADS juad namery
0.70-
PPC
wo005osel-eeee neneyt
0.00-
0.30
0.90-
-0.55
4025-
-066-
-0.00
-000-
-0.76
-0.79
-0.7-
-1.00
250
500
713
1900
1250
1300
1750
2000
-1.00
22:
300
1h0
1000
1250
1250
1750 2000
0
250
500
153
1000
1250
1300 1750 2000
Genoration
Goneration
Generation
(a)
(b)
(c)
Figure 6: The Only-History Challenge: Training curves in the IMP with episode length = for (a) CHAOS and (b) CHAOS without context. Note that in short time-spans, history can not be used context can enable shaping. Additionally (c) CHAOS without context in IMP with episode length 100 shows with sufficient timespan, history can be used to shape.
cooperating (CC) or exploiting their co-players are the most likely states observed during training (50% and 20% respectively).
exploit co-players (DC), here see these two
We found GS produces comparable results to CHAOS. At first this is surprising, since GS is feedforward network and does not have access to the history. Therefore, in principle it should not be able retaliate against de- fecting agents since it has no memory of their past actions. However, close investigation of the problem setting shows that due to the spe- cific environment dynamies, the current state is often indicative of past actions. For example, when the two agents are on the same square, in all likelihood one of the agents defected (see Figure D Similarly, if the agents are currently standing a on coin of given colour, this coin was picked up the last time step. This illustrates that Coin Game allows for simple shaping strategies that do not require context or history, limiting its utility as a benchmark for investigating these aspects.
Marbbuu Challargu (T-21-2)
Harbbto Cha PC; (T,(7-2)
22-
-2-
-22
25
-12
Epitado
The Hardstop Challenge: Average re- ward over a single trial for CHAÓS (left) and GS (right) against a naive learner in the IPD.
Figure 7:
Ablations CHAOS outperforms GS on the hardstop challenge. CHAOS demonstrates dynamic shaping by switching strategies at t = 2 episodes, when the hardstop is triggered (see Fig. ??). In contrast, GS's policy is fixed throughout a trial and thus can not exploit the hardstop.
In the Only History challenge, when playing the IMP with a small number of inner-episodes (E- 2), we expect shapers without history to be unable to identify co-players current learning and thus be unable to shape. We find that CHAOS capable of this, whilst CHAOS( without history) is unable to shape agents 6 Interestingly, we also found that with a longer inner-episode length (E' = 100), CHAOS is able to use context to shape its co-player [oc This show that both context and history are required over different environments to encode co-players learning.
7 CONCLUSION
In this paper, we introduce CHAOS, a shaping method capturing both learning context and history, suitable for nigh-dimensional games. We formalise the concept of history and context for shaping and analyse their respective roles empirically. We demonstrate state-of-the-art performance on set of iterated matrix games. We identify a fundamental problem in the widely-used Coin Game.
When multiple agents interact in a shared environment, the actions of each any one agent influ- ence the rewards and environment faced by others, and through their learning ultimately affect their behaviour. Shaping, i.e. constructing agents that can effectively leverage this interconnection. has emerged as a sub-field in Mult-agent Reinforcement learning and has received considerable attention in recent years. CHAOS substantially expands the current capabilities of shaping agents by allowing them to react to changes to the co- co-players' learning dynamics as well as to predictable patterns in their within-episode behaviour, thus resulting in significantly more effective shaping.

---

Under review as a conference paper at ICLR 2023
8 ETHICS STATEMENT
Shaping can be for good and bad. Empirically, shaping has lead AIs to find more prosocial solutions. However, one imagine scenarios where shaping be used with negative impact society- Investigating shaping is important to prevent misuse of the paradigm. We are at the beginning of fundamental research in shaping and better understanding the necessary components shaping will help us to better control shaping agents. Opponent Shaping is still early in an early phase of development and practical implications are limited so immediate negative societal infuence is unlikely.
achieve
REFERENCES
Maruan Al-Shedivat, Trapit Bansal, Yura Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. In In- ternational Conference on Learning Representations, 2018. 2
Robert Axelrod and William D Hamilton. The evolution of cooperation. science, 21 114(449)) 1396, 1981. 1
Jan Balaguer, Raphael Koster, Christopher Summerfield, and Andrea Tacchetti. The good shepherd: An oracle agent for mechanism design. arXiv preprint arXiv:2202.10133, 2022. 12 6
David Balduzzi, Sébastien Racanière, James Martens, Jakob N. Foerster, Karl Tuyls, and Thore Graepel. The mechanics of n-player differentiable games. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.
363-372, 2018. 6
James Bradbury, Roy Frostig. Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnee and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http: //github. httpbb/ccm/oogge///eoeoo
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RI$-2S: Fast reinforcement learning via slow reinforcement learning. arXiv preprint 1611v:1611.02777,
2016.
2 3
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 112--1135, 2017. 2
Jakob Foerster, Richard Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor
Mordatch. Learning with opponent-leaming awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 122-130, 2018.
2 6
Jakob Foerster, Francis Song, Edward Hughes, Neil Burch, Iain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling. Bayesian action decoder for deep multi-agent re- inforcement learning. International Conference on Machine Learning, pp. 1942-1951. PMLR,
2019. 1
David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetorkss. In 5th International Conference on Learning Representations, 2017.
Mare Harper, Vincent Knight, Martin Jones, Georgios Koutsovoulos, Nikoleta E. Glynatsi, and Owen Campbell. Reinforcement learning produces dominant strategies for the iterated prisoner's dilemma. PLOS ONE, 12(12):e0188046 2017.
Tom Hennigan, Trevor Cai, Tamara Norman, and Igor Babuschkin. Haiku: Sonnet for JAX, 2020. URL httt::///tttbb.. com/ cttm:/deeminni nd/ dm-haiku
Dong-Kr Ki Kim, Miao Liu, Matthew Riemer, Chuangchuang Sun, Marwa Abdulhai, Golnaz Habibi, Sebastian Lopez-Cot, Gerald Tesauro, and Jonathan P. How. A policy gradient algorithm for learning to lear in multiagent reinforcement learning. In International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 5541-5550 2021a.
10

---

Under review as a conference paper at ICLR 2023
Dong Ki Kim, Miao Liu, Matthew D Riemer, Chuangchuang Sun, Marwa Abdulhai, Golnaz Habibi, Sebastian Lopez- Copez-Cot, Gerald Tesauro, and Jonathan How. A policy gradient algorithm for learning to learn in multiagent reinforcement learning. In International Conference Machine Learning, pp. 5541-5550. PMIR, 2021b. 2
Robert Tjarko Lange. evosax: Jax-based evolution strategies, 2022. URL http://github.
Con//obbertlaange/eeee
Adam Lerer and Alexander Peysakhovich. Maintaining cooperation in complex social dilemmas using deep reinforcement learning. CoRR, abs/1707.01068. 2017. 6
Alistair Letcher, David Balduzzi, Sébastien Racanière, James Martens, Jakob N. Foerster, Karl Tuyls, and Thore Graepel. Differentiable game mechanics. J. Mach. Learn. Res., 20:84:1-84:40, 2019a. 2
Alistair Letcher, Jakob N. Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson. Stable opponent shaping in differentiable games. In 7th International Conference on Learning Repre- sentations, 2019b.
Christopher Lu, Timon Willi, Christian A Schroeder De Witt, and Jakob Foerster. Model-free op- ponent shaping. International Conference on Machine Learning. pp. 14398-1444 14411. PMLR, 2022.
L
Richard Mealing and Jonathan I.. Shapiro. Opponent modeling by expectation-maxiiizaaioo and sequence prediction in simplified poker. IEEE Trans. Comput. Intell. AI Games, 91):11-24, 2017. 2
William H. Press and Freeman J. Dyson. Iterated Prisoner's Dilemma contains strategies that dom- inate any evolutionary opponent. Proceedings of the National Academy of Sciences, 109(26): 10409-10413, 2012. ISSN 0027-8424. 6
Roberta Raileanu, Emily Denton, Arthur Szlam, and Rob Fergus. Modeling others using oneself in multi-agent reinforcement learning. In International conference on machine learning, pp. 4257- 4266. PMLR, 2018. 2
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforce- ment learning. International conference on machine learning, pp.
PMLR, 2018.
Tim Salimans, Jonathan Ho, Xi Chen, and Ilya Sutskever. Evolution strategies as a scalable alterna-
tive reinforcement learning, arXiv preprint arXiv: wet:::1030088666 2017. 2 5
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017.
optimization algorithms. arXiv preprint
2
7
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driess- che, Julian Schrittwieser, loannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nat., 529(7587):484 489, 2016. [
Gabriel Synnaeve and Pierre Bessière. A bayesian model for opening prediction in RTS games with application starcraft. In IEEE Conference Computational Intelligence and Games, pp- 281-288, 201 1.
Andrea Tacchetti, H Francis Song. Pedro AM Mediano, Vinicius Zambaldi, Neil Rabinowitz, Thore Graepel, Matthew Botvinick, and Peter W Battaglia. Relational forward models for multi- agent learning. arXiv preprint 1rXiv:1809. /044, 2018.
Timon Willi, Alistair Letcher, Johannes Treutlein, and Jakob N. Foerster. COLA: consistent learning with opponent-learning awareness. In International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 23804-23831 2022.
Zhe Wu, Kai Li, Enmin Zhao, Hang Xu, Meng Zhang, Haobo Fu, Bo An, and Junliang Xing. L2E: learning exploit your opponent. arXiv preprint arXiv:2102.09381. 2021. 2

---

Under review as a conference paper at ICLR 2023
A MATRIX GAME DETAILS
Here present details of training of shaping agents in Iterated Matrix Games.
A.1 PAYOFF MATRICESS
Table 2: Payoff Matrices
C
D
H
(-1,-1)
-0)
(1,- 1)
(-1, 1)
D
(-3, 0)
(-2, -2)
(-1 I)
(a) Iterated Prisoners Dilemma (IPD)
(b) Iterated Matching Pennies (IMP)
A.2 TRAINING DETAILS
We present training curves for both IPD and IMP below.
Training CHAOS vs PPO
Probabilily Cooperation
State Visitation
00-
State Vaitation Frequence
CHA06
-30-
PPG
RCConerraton)
0.8
0.8
Filness
-1.
0.6-
0.6
START
START
0.4
-2.5
0.-
-30
1000
1800
2500
0.0
1000 Generation
1000
2000
2000
2000
0.0
000
1000
1000 Generation
2000
2000
Generation
(a)
(b)
(c)
1.00
CHAD6
Training CHADS vs PPO
1.0
HH
Probability of Heads
State Visitaiion Frequeeccy
1.0
H#1
State Visitation
FPO
HT
HT
PiHeads)
TH
0.20
0.00
-0.25
0.4
0.4
02-
-0.70
1000
2000
3000
4000
0000
1000
2000
3000
4000
5000
1000
2000
3000
4000
5900
Generation
Generation
Ganeration
(d)
(e)
(I)
Figure 8: Training results in the finite IPD seeds for CHAOS. Here display the (a) fitness, (b) conditional probability of cooperation, and (c) state visitation. Training results in the IMP over seeds for CHAOS. (d) Fitness (c) Empirical probability of Cooperative action conditioned by state and (f) state visitation.
B COIN GAME DETAILS
B. EXPERIMENTAL PROTOCOL
We define an experimental protocol for producing the correct conditions for shaping
Experimental Protocol
We verify that an agent can learn against three competent opponents. The first agent is Good Greedy, picking up coins indiscriminately but prioritizing its own colour coin if it is equidistant from two coins; the second agent is Evil Greedy, picking up coins indiscriminately but prioritizing its opponent's 's coin is equidistant from two coins; the third agent is the current agent pre-trained to competency via self-play. After verifying that an agent learns competency against these three opponents, we set the parameters of the trial to reflect the time and scale required for the agent become competent against competent opponent.
12

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(c)
(d)
Figure 9: Illustration of (a) state observed by both agents in the non-egocentric Coin Game and (b)-(c) the same state observed by the blue and orange agent, respectively, in the egocentric Coin Game. (d) Illustration of special state where memory-less agents can infer conventions.
Sanity We verify that PPO RNN can learn against competent opponents the Coin Game in Figure 10 PPO RNN learns to pick up the same number of coins pre-trained PPO RNN, Good Greedy, and Evil Greedy. These hyperparameters are the same ones used during meta-leamiig.
Coin Game Adjustments
In the Coin Game, agents struggle to learn (via reinforcement learning) when trained against pre- trained opponent. On inspection of trajectories, we found that competent agents removed a sufficient amount of coins to restrict reinforcement learners ability to capture signal from the game.
To address this, we show that adjusting the observations such that agent receives to egocentric viewpoint (i.c. an agent always observes that in the centre of the grid) leads to competency against competent opponent. In this case, measured competency up coins. Competent agents were those who picked up a similar number of coins to those trained against stationary agent. Throughout the rest of the paper, we refer to the original version of Coin Game as non-egocentric Coin Game and the modified observation version as egocentric Coin Game. In addition, we deviate from the original by 5 version of Coin Game to 3 by version, following the setting used in (Lu et al. 2022
agent's ability to pick
Average Coins Collected Episode
Average Coins Collected Per Episode
Average
Episode
16.
16
14-
Cons
Coins
10
Coims
PMD RNN
PFU RNN
PPU RNN
Pre-taiired PPO RNN
Good Greedy
Evit Greedy
8000
20.0
10000
4000 5002
10000
Episode
Episode
Episode
(c) Figure 10: Results of the experimental protocol verifying that PPO RNN learns play the egocentric Coin Game against (a) pre-trained PPO RNN (b) Good Greedy (c) Evil Greedy. Notice that the agent learns to pick up roughly the same number of coins per episode as its competent opponents.
(a)
(b)
B.2 TRAINING DETAILS
B.3
GENERALIZATION IN PRE-TRAINED AGENTS
C ABLATION STUDIES
We also include additional state visitation for the hardstop challenge. This helps validate that CHAOS reacts agents becoming exploitable after hardstop has occurred.
13

---

Under review as a conference paper at ICLR 2023
190
Tresing CHAO0S 1905
Probabiltty
14
Coins Picked Up
Probetely of Exoperatin
7.
FPORMM
50
73
20
600
12la
R02
202
030
10
80 N
Cosserddion
Gommentoo
Canenation
Conmenton
(a)
(b)
(c)
(d)
Figure 11: Training results of CHAOS vs. PPO RNN the egocentric Coin Game. (a) Fitness, (b) the shaper's frequency of picking up its own colour coin depending on existing convention, (c) the number of coins picked up per episode, (d) both agent's 's frequency of picking its own colour coin over full episode.
EARL pre-trained on Tabular
EARL pre pre-frained on PPO RNN
EARL pre trained on PPO
10.0
100
10.0
60-
60-
6:0-
Roward
Reward
0.-
00-
0.--
-2.5
-50-
-50
LVB.
-7.-
-7.-
-100
100
-100
4000
6000
1000
2000
0000
0000
10000
2000
0000
10000
Episode
Episdde
Episode
(c)
Figure 12: Results of training versus a CHAOS agent pre-trained against (a) Tabular, (b) PPO RNN, and (c) PPO. Notice that the CHAOS agent pre-trained against Tabular outperforms the CHAOS agent pre-trained against PPO RNN. We hypothesize that CHAOS finds "true" shaping against the Tabular agent but sub-optimal shaping policy against PPO RNN, due to its complexity. Table ?? shows that CHAOS outperforms PPO RNN (7.08, 77.66 by wider margin than against Tabular (6.67.6.55) 6651), indicating that CHAOS finds shaping policy that performs well against PPO RNN, but that this shaping policy does not generalize other agents robustly the shaping policy learned against Tabular.
Hardstop Challenge, CHAOS, (T=2)
Hardstop Challenge, GS, (T=2)
State Frequency
State Frequency
0.6
CC
CC
CD
CD
DC
DC
DD
DD
START
START
80
20
80
100
Episode
Episode
Figure 13: State visitation for the Hardstop Challenge with shapers (a) CHAOS and (b) GS. Here see CHAOS responds to co-players frozen stationary policy moving into either DD (the best response to a defective agent) or DC (the best response to a fully cooperative agent), whilst GS is unable to adjust its approach after the co-player stop learning, continuing to plays the sub-optimal shaping strategy)
14

---

Under review as a conference paper at ICLR 2023
D HYPER-PARAMETERS
Hyperparameter
Value
Number of Actor Hidden Layers
Number of Critic Hidden Layers
Torso GRU Size
[25]
Length of Meia-Ep
100
Length of Inner Episode
100
Number of Generations
5000
Batch Size
100
Population Size
1000
OpenES sigma init
0.04
OpenES sigma decay
0.999
OpenES sigma limit
0.01
OpenES init min
0.0
OpenES init max
0.0
OpenES clip min
-1e10
OpenES clip max
le10
OpenES Irate init
0.01
OpenES Irate decay
0.9999
OpenES Irate limit
0.001
OpenES beta
0.99
OpenES beta
0.999
OpenES eps
1e-8
Table 3: Hyperparameters for CHAOS in Iterated Prisoner's Dilemma
Hyperparameter
Value
Number of Minibatches
4
Number of Epochs
2
Gamma
0.96
GAE Lambda
0.95
PPP clipping epsilon
0.2
Value Coefficient
0.5
Clip Value
True
Max Gradient Norm
0.5
Entropy Coefficient Start
0.02
Entropy Coefficient Horizon
2000000
Entropy Coefficient End
0.001
Learning rate
1
ADAM epsilon
le-5
Table 4: Hyperparameters for Tabular-PPO in Iterated Prisoner's 's Dilemma
15

---

Under review as a conference paper at ICLR 2023
Hyperparameter
Value
Number of Actor Hidden Layers
2
Number of Critic Hidden Layers
2
Network Hidden Size
[16, 16]
Length of Meta- Seta-Episode
100
Length of Inner Episode
100
Number of Generations
5000
Batch Size
100
Population Size
1000
OpenES sigma init
0.04
OpenES sigma decay
0.999
OpenES sigma limit
0.01
OpenES init min
0.0
OpenES init max
0.0
OpenES clip min
-le10
OpenES clip max
le10
OpenES Irate init
0,01
OpenES Irate decay
0.9999
OpenES Irate limit
0.001
OpenES beta I
0.99
OpenES beta 2
0.999
OpenES eps
1e-8
Table 5: Hyperparameters for GS in Iterated Prisoner's Dilemma
Hyperparameter
Value
Number of Actor Hidden Layers
Number of Critic Hidden Layers
Actor GRU Hidden Size
16
Critic GRU Hidden Size
Meta Agent Gru Hidden Size
16
Hidden Layer Size
Length of Meta-Episode
100
Length of Inner Episode
100
Number of Generations
5000
Batch Size
100
Population Size
1000
OpenES sigma init
0.04
OpenES sigma decay
0.999
OpenES sigma limit
0.01
OpenES init min
0.0
OpenES init max
0.0
OpenES clip min
-le10
OpenES clip max
le10
OpenES Irate init
0.01
OpenES Irate decay
0.9999
OpenES Irate limit
0.001
OpenES beta
0.99
OpenES beta
0.999
OpenES eps
1e-8
Table 6: Hyperparameters for MFOS in Iterated Prisoner's Dilemma
16

---

Under review as a conference paper at ICLR 2023
Hyperparameter
Value
Number of Actor Hidden Layers
Number of Critic Hidden Layers
Torso Gru Size
[16]
Length of Meta-Episode
600
Length of Inner Episode
16
Number of Generations
3000
Batch Size
100
Population Size
4000
OpenES sigma init
0.04
OpenES sigma decay
0.999
OpenES sigma limit
0.01
OpenES init min
0.0
OpenES init max
0.0
OpenES clip min
-1e10
OpenES clip max
lc10
OpenES Irate init
0.01
OpenES Irate decay
0.9999
OpenES Irate limit
0.001
OpenES beta
0.99
OpenES beta
0.999
OpenES eps
1e-8
Table 7: Hyperparameters for EARL in Iterated Matching Pennies
Hyperparameter
Value
Number of Minibatches
8
Number of Epochs
2
Gamma
0.96
GAE Lambda
0.95
PPO clipping epsilon
0.2
Value Coefficient
0.5
Clip Value
True
Max Gradient Norm
0.5
Anneal Entropy
False
Entropy Coefficient Start
0.02
Entropy Coefficient Horizon
2000000
Entropy Coefficient End
0.001
LR Scheduling
False
Learning Rate
0.005
ADAM Epsilon
lc-5
With CNN
False
Table 8: Hyperparameters for PPO Memory and Tabular in the Coin Game
17

---

Under review as a conference paper at ICLR 2023
Hyperparameter
Value
Number of Actor Hidden Layers
Number of Critic Hidden Layers
Hidden Size
[16]
Length of Meta-Episode
600
Length of Inner Episode
16
Number of Generations
3000
Batch Size
100
Population Size
4000
OpenES sigma init
0.04
OpenES sigma decay
0.999
OpenES sigma limit
0.01
OpenES init min
0.0
OpenES init max
0.0
OpenES clip min
-1e10
OpenES clip max
lc10
OpenES Irate init
0.01
OpenES Irate decay
0.9999
OpenES Irate limit
0.001
OpenES beta
0.99
OpenES beta
0.999
OpenES eps
le-8
Table Hyperparameters for GS Coin Game
Hyperparameter
Value
Number of Actor Hidden Layers
Number of Critic Hidden Layers
Actor GRU Hidden Size
16
Critic GRU Hidden Size
16
Meta Agent Gru Hidden Size
16
Hidden Layer Size
Length of Meta-Episode
100
Length of Inner Episode
100
Number of Generations
5000
Batch Size
100
Population Size
1000
OpenES sigma init
0.04
OpenES sigma decay
0.999
OpenES sigma limit
0.01
OpenES init min
0.0
OpenES init max
0.0
OpenES clip min
-1e10
OpenES clip max
1e10
OpenES Irate init
0.01
OpenES Irate decay
0.9999
OpenES Irate limit
0.001
OpenES beta
0.99
OpenES beta
0.999
OpenES eps
1e-8
Table 10: Hyperparameters for MFOS in Coin Game
18


Reviews:
[{"review_id": "x3kxs9sLZLU", "reviewer": "Reviewer_PvhL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an algorithm, called CHAOS (Context and History Aware Other-Shaping) with the goal of capturing both learning context and history.  ", "summary_of_the_review": "N/A\n\n ", "strength_and_weaknesses": "Strength: \n- Some empirical results. \n\nWeaknesses:\n- The novelty is unclear.\n- Presentation lacks clarity and precision. The theory background is incomplete and vague, also contains several mistakes/undefined parts. \n- No actual result is presented beyond some empirical evaluation, which is neither conclusive nor sufficient. \n- No formal reasoning or meaningful argument is given. I am not sure what the usefulness of this work would be at all for the ICLR audience.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aBYTfRET-_", "reviewer": "Reviewer_Ae1h", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates training an agent that's own learning algorithm attempts to \"shape\" how its' co-players learn. This shaping ideally should cause it so that the co-players' play an equilibrium that is favorable to the agent employing the shaping. This technique was previously studied in the Model-Free Opponent Shaping paper referred to in the abstract. They show that a policy trained through evolutionary strategies is able to employ such a co-player shaping policy. ", "summary_of_the_review": "I do not think the paper as it currently exists is ready for publication. I am not convinced that this work is significantly different enough from the previous work to justify publication (it appears to just be allowing a RNN's state to persist across episode). Additionally, I found the prose rather challenging to follow and the claims and method not clear enough to contribute to the scientific discussion of the area.", "strength_and_weaknesses": "**Strengths**\n - The authors demonstrate that memory persistent across episodes does not confound their results, which I suspected during reading the first portion of the paper.\n \n**Weaknesses**\n - This work appears to have significant overlap with the prior work M-FOS to the point where I'm not sure if it is different. Inspection of the Algorithm block from this paper and the previous are almost identical. I believe the CHAOS one also contains a typo and that the phi on line 13 should be theta. Moreover, in Section 4 of the M-FOS paper they directly mention that genetic algorithms (a type of evolutionary strategy) would work well within this framework. \n - The method section is challenging to follow and not technical enough to be confident in how the algorithm actually works. I would encourage the authors to add exact optimization functions and how they fit within the overall algorithm. \n - If it is true that the previous work cannot handle settings where DNN implement policies, a statement I am not convinced of from this manuscript, I would like to have seen a demonstration of this algorithm in such a setting and explanation as to why the previous work could not handle it. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SeY_fofGTMk", "reviewer": "Reviewer_Hqnc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a meta-learning approach to shaping the policy of other agents in a multi-agent learning setting. The proposed method, called CHAOS, meta-trains an RNN agent policy that utilizes both intra-episode history and inter-episode context information to select actions so that it can successfully shape the policies learned by the other agents to maximize its own rewards. Experiments are performed on 2-player matrix games to show that CHAOS outperforms or matches previously proposed meta-learning based other shaping algorithms that become computationally expensive with increasing action space dimensions or that do not utilize context and history information in their policy. \n", "summary_of_the_review": "I recommend rejection of the paper in its current form. The description of the proposed method and the implementation and evaluation details need to be further elaborated and the experimental results should address the questions I have raised above.\n", "strength_and_weaknesses": "The paper contextualizes the prior work and outlines the proposed improvements compared to the related approaches M-FOS and GS. The following points require further correction / clarification. \n \n1) There are some typos throughout the paper.\n- Sec 3, POSG: reward function is $\\mathcal{R}$ and not $\\mathcal{\\tau}$? \n- Sec 3, POSG: single player case $\\mathcal{I} = \\{1\\}$ or $N = \\{1\\}$?\n- Page 4, Good Shepherd: The gradient is wrt $\\phi^e_{-i}$?\n- Page 4, Good Shepherd, last sentence in the paragraph is incorrectly framed?\n- Sec 4, paragraph 2: Inconsistent notation in policy ($\\phi$, $\\theta$) and distribution ($\\rho$). \n- Sec 5.3: “...1)identifying the … an co-player’s …” -> “..identify…. a co-player’s…”\n2) Algorithm 1: Line 9 - is the naive learner updated every time step or after T steps in the trial? There is no mention of E episodes. How are $\\theta_m$ and $\\theta_n$ different? Line 13 - update $\\phi$ or update $\\theta_m$ or $\\theta_n$?\n3) It would help to clearly describe the evaluation setting in the meta-learning framework. Are the results reported for 1 trial after the training has converged for both the meta-learner and the opponent? Is the opponent being trained from scratch during the evaluation trial? Is the opponent reset / re-initialized at the beginning of each trial during training as well as evaluation?\n4) It would help to clearly describe the legends in subfigures showing probability of cooperation conditioned by state and the state visitation frequency. Particularly, it was unclear to me, for example in Fig 2b whether CC is the action of the (opponent, meta-learner) or (meta-learner, opponent) in the previous step? What are the corresponding states in Fig 5b / 5e / 5f ?\n5) Sec 6, Coin game: “Both GS and CHAOS demonstrate shaping … (see Fig 5).” - I did not understand this, needs further explanation.\n6) Sec 6, Ablations: “.. when the hardstop is triggered (see Fig 6c).” - I don’t think this is shown in Fig 6c since it contradicts the description in the caption for Fig 6c. \n7) Fig 6: Is the single trial evaluated after the training has converged for both CHAOS and PPO? \n8) The x-axis has a different range in Fig 6d compared to Fig 6c and 6e - it does not result in a fair comparison. The plot for ‘PPO’ should be the same in Fig 6c and Fig 6d and maybe we will see this if the range / scale of the x-axis is fixed. The paper says that in contrast with Fig 6d, Fig 6e shows with sufficient timespans, history can be used to shape in the absence of context  - but the curves are almost identical till 200 generations in Fig 6e as well as Fig 6d. \n9) Why are the curves for PPO different in Fig 6a and Fig 6b? Are they comparable? Can you show the state visitation frequency with CHAOS, GS and PPO for this ablation study?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "73K4smtpQY", "reviewer": "Reviewer_zKsT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to capture co-playr learning dynamics in MARL. The authors propose Context and History Aware Other-Shaing (CHAOS) to address this problem. The CHAOS agent is a meta-learner using RNN architecture to learns to shape its co-player. The authors conduct extensive experiments on matrix games.", "summary_of_the_review": "My main concern for the paper is its novelty and experimental evaluation. It would be better to clarify more details and motivation for proposing the algorithm in Section 4. It can also be improved by evaluating CHAOS in more commonly-used MARL benchmark such as SMAC by comparing it with recent MARL algorithms. ", "strength_and_weaknesses": "One of the main conerns for the paper is its novelty. The CHAOS algorithms seems to be a combination of the existing components and methods introduced in the background section. Can the author better clarity the novelty of CHAOs? \n\nAnother concern for the paper is its experimental part. \n- It is claimed that the proposed method (CHAOS) is suitable for high-dimensional games. However, the most complex experimental benchmark in this paper is the Coin Game in a grid-world style. It is far less complex than the commonly used, challenging, and high-dimensional SMAC benchmark. It is therefore worth evaluating the method on more complex benchmark to better support the claim.\n- It is also mentioned that \"Cooperation failures, in which self-interested agents converge to collectively worst-case outcomes, are a common failure mode of MARL methods.\" Is this problem common for widely-used MARL algorithms like QMIX and MAPPO. In addition, how does the method compare against these popular baselines?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject