Opponent Shaping in LLM Agents
Authors: Marta Emili García Segura
Stephen Hailes
Mirco Musolesi
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Agents
Opponent Shaping
Multi-agent Systems
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Large Language Models (LLMs) are increasingly being deployed as autonomous agents in real-world environments. As these deployments scale, multi-agent interactions become inevitable, making it essential to understand strategic behavior in such systems. A central open question is whether LLM agents, like reinforcement learning agents, can shape the learning dynamics and influence the behavior of others through interaction alone. In this paper, we present the first investigation of opponent shaping (OS) with LLM-based agents. Existing OS algorithms cannot be directly applied to LLMs, as they require higher-order derivatives, face scalability constraints, or depend on architectural components that are absent in transformers. To address this gap, we introduce ShapeLLM, an adaptation of model-free OS methods tailored for transformer-based agents. Using ShapeLLM, we examine whether LLM agents can influence co-players’ learning dynamics across diverse game-theoretic environments. We demonstrate that LLM agents can successfully guide opponents toward exploitable equilibria in competitive games (Iterated Prisoner’s Dilemma, Matching Pennies, and Chicken) and promote coordination and improve collective welfare in cooperative games (Iterated Stag Hunt and a cooperative version of the Prisoner’s Dilemma). Our findings show that LLM agents can both shape and be shaped through interaction, establishing opponent shaping as a key dimension of multi-agent LLM research.

Published as a conference paper at ICLR 2026
OPPONENT SHAPING IN LLM AGENTS
Marta Emili Garcia Segura"? Stephen Hailes', Mirco Musolesi
'Department of Computer Science, University College London Centre for Artificial Intelligence, University College London DDepartment of Computer Science, University of Bologna {marte autara.segura.... segura. 22, s s.hailes, m. m.muo mmusolllsi@@uul.... www.llesselllc........o ac..uk
ABSTRACT
Large Language Models (LLMs) are increasingly being deployed as autonomous agents in real-world environments. As these deployments scale, multi-agent inter- actions become inevitable, making it essential to understand strategic behavior in such systems. A central open question is whether I.L.M agents, like reinforcement learning agents, can shape the learning dynamics and influence the behavior of others through interaction alone. In this paper, we present the first investigation opponent shaping (OS) with LLM-based agents. Existing OS algorithms cannot be directly applied to LLMs, as they require higher-order derivatives, face scala- bility constraints, depend architectural components that are absent in trans- formers. To address this gap, we introduce ShapeLLMM, an adaptation of model- free OS methods tailored for transformer-based agents. Using ShapeI.LM, we examine whether LLM agents can influence co-players' learning dynamics across diverse theorrtc environments. We demonstrate that LLM agents can suc- cessfully guide opponents toward exploitable equilibria in competitive games (It- erated Prisoner's Dilemma, Matching Pennies, and Chicken) and promote coor- dination and improve collective welfare in cooperative games (Iterated Stag Hunt and cooperative version of the Prisoner's Dilemma). Our findings show that LLM agents can both shape and be shaped through interaction, establishing oppo- nent shaping as a key dimension of multi-agent LLM research.
INTRODUCTION
Large language models (LLMs) have evolved rapidly in recent years, demonstrating remarkable ca- pabilities in reasoning, planning and goal-directed behavior that make them increasingly suitable for deployment as autonomous agents (Zhao et al. 2023 Anthropic 2025a OpenAl (025b) Xi et al., 2025). Already, LLM-based agents are being adopted for complex tasks such as web navigation and code generation (Anthropic 20256 OpenAl 2025a). As deployment scales, these agents will be less likely to operate in isolation. Instead, they will increasingly interact with other learning agents in shared environments, collaborating tasks, competing for resources, pursuing independent objectives. There is growing interest in understanding the opportunities and challenges associated with multi-agent LLM systems (Fourney et al. [2024] Ghafarollahi & Buchler 2025 Pan al. 2025 Rosser & Foerster 2025). However, most approaches treat LLMs as static entities, overlooking the strategic dynamies that emerge when agents continuously adapt one another.
Multi-agent reinforcement learning (MARL) has long been concerned with the interaction of multi- ple learners in shared environments (Busoniu al., (008). A core difficulty MARL is that agents often treat each other as static parts of the environment, which can yield poor collective outcomes. For instance, the Iterated Prisoner's Dilemma (IPD, Axelrod Hamilton ( (198T)) independent learners reliably converge to mutual defection, which is the worst collective outcome (Harper et al.) 2017 Foerster et al. 2018). To mitigate such failures, the field of opponent shaping develops agents that actively anticipate and influence their co-players' learning dynamics, steering learned behavior toward more favorable equilibria (Foerster et al.) 2018 Lu et al. 2022). While these methods have proven effective for multiple architectures (e.g., tabular policies and recurrent neural networks), it remains unclear whether they extend to LLM agents. These agents process rich semantic informa- tion, exhibit complex reasoning capabilities (Xu et al. 2025), and can adapt their behavior through

---

Published as a conference paper at ICLR 2026
in-context learning (Brown al. 2020; Bubeck et al. 2003). This raises uncertainty about whether traditional shaping methods will transfer to LLM agents.
Our work addresses this gap. being the first exploration of opponent shaping with LLM agents. Understanding the extent to which LLMs can engage in opponent shaping is critical as they are in- creasingly deployed in real-world, multi-agent settings. This capability has dual implications: LLM agents may be vulnerable to exploitation by adversaries who strategically influence their learning dynamics, while shaping could also be tool foster prosocial behavior and enable coordination.
We study whether LLM agents can strategically influence each other's learning dynamics in repeated matrix games, which capture core strategic incentives while allowing precise outcome quantification. As a baseline, we train two LLM agents independently using Proximal Policy Optimization (PPO, Schulman al. 2017)) to maximize their individual returns. We then introduce opponent shaping by turning one of those agents into shaper that aims to alter the learning dynamics of its co-player. The shaper is trained using ShapeLL.M, our proposed algorithm that adapts model-free approaches (Lu et al., [222; Khan et al.) (2044) to transformer architectures. We evaluate the efficacy of shaping in both exploitative scenarios, where an agent seeks unilateral advantage, and prosocial scenarios, where shaping fosters cooperation. Our contributions are the following:
We provide the first investigation of opponent shaping in LLM agents, demonstrating that they can strategically influence each other's learning dynamics through interaction alone.
We propose ShapeLLM, model-free opponent shaping algorithm for transformer archi- tectures leveraging structured natural language prompts.
We evaluate shaping across diverse game-theoretic environments and show that LLM agents can successfully exploit opponents in competitive settings and guide interactions toward mutually beneficial outcomes in cooperative ones.
BACKGROUND
2.1 LLM AGENTS
An LLM agent can be thought of as any system leveraging LLMs as the core computational unit
for reasoning, planning, and decision-making (Sumers et al. 2023). The architectures of these agents vary significantly in the literature, with some systems integrating reasoning frameworks (Yao
iet al.] 2023), memory banks /Vezhnevets et al. 2023). or tool-access via APIs (Schick et al.
2023
Patil al. 2024), both in single and multi-agent settings (Park et al. 2023 Wang al. 2024).
In the latter, game-theoretic environments provide a natural testbed for studying strategic dy dynamics.
Recent work has used these environments investigate LLM agents' cooperation (Piatti al. 2024
Akata et al. 2025). rationality (Fan et al. 2024). and strategic reasoning (Gandhi et al. 2023 Duan
Huang al., 2025). Beyond observational studies, these environments can be used to train LLMs towards specific objectives such as moral aligmment (Tennant et al.) (025). In this work,
al.) 2024
we employ game-theoretic environments as controlled testbeds with quantifiable outcomes and clear incentive structures study whether LLM agents can both shape and be shaped by opponents.
2.2 FINE-TUNING LLMS WITH REINFORCEMENT LEARNING
The use of reinforcement learning (RL) in LLM training was first popularized through Reinforce-
ment Learning from Human Feedback (RLHF, Ziegler et al. (2019); Stiennon et al. (2020): Ouyang al. (2022)) which typically employs Proximal Policy Optimization (PPO.(Schulman et al. (2017) )
as the RL algorithm. PPO is an on-policy, actor-critic method that uses a learned value function for advantage estimation. When applied to L.Ms, it is customary to include Kullhack-Leible (KL) penalty in the reward signal to prevent the model's output distribution from diverging too far
from the pre-trained one. Several alternatives to RLHF with PPO have been proposed, including Group Relative Policy Optimization (GRPO, Shao et al. (0024)), which estimates advantages via Monte Carlo rollouts, and Direct Preference Optimization (DPO, Rafailov et al. (2024)), which con- verts the RLHF objective into supervised learning loss. All three approaches
typically used in contextual bandit settings (Llama Team, AI@ Metal 2024at Qwen Team 2025) Mistral-A]] (025), where each model response is treated an independent episode with immediate reward feedback.
2

---

Published as a conference paper at ICLR 2026
The application of multi-turn RL to LLMs remains active area of research due to challenges in preference collection, reward modeling, and ambiguity in action space definition (Shani et al.) 2024 Zhou et al., 2024, Zeng et al. 2025 Nevertheless, underlying algorithms such as PPO are inherently designed to handle temporally structured environments.
2.3
OPPONENT SHAPING
The opponent shaping literature is characterized by two primary approaches: methods that explicitly account for the opponent's ' updates in the agent's learning rule (Foerster et al. 2018 Letcher et al.,
2019
Willi et al. 2022), and meta-learning approaches that learn to shape opponents by observing
how their actions influence their opponent's ' parameter updates (Lu et al.) 2022, Balaguer et al.
2022 Khan et al. 2024). The most notable method in the first category is Learning with Opponent- Learning Awareness (LOLA, Foerster et al. (2188). ). In LOLA, the opponent's s learning rule is incorporated into the agent's 's update, accounting for the effect of the agent's 's action on the opponent's parameter updates. This method has demonstrated notable successes, such the emergence of Tit- for-Tat (TFT) in the iterated prisoner's dilemma (IPD) through self-play. However, LOLA faces significant limitations: it assumes knowledge of the opponent's 's learning rule, relies high-variance higher order derivatives, and only considers immediate effects on the opponent's 's updates. While several LOLA refinements have been proposed (Letcher al. 2019 Willi al., (022). they exhibit the same limitations. A distinct line of work proposes an opponent shaping algorithm based on modified advantage function which includes the opponent's 's advantage (Duque et al. (025). While being higher-order-derivaivv free, this approach requires access to the opponent's value function, distinguishing from fully model-free methods.
The second category of methods, exemplified by Model-Free Opponent Shaping (M-FOS, Lu et al. p0222), ) was developed specifically to overcome these challenges. M-FOS bypasses these limi- tations, particularly LOLA's myopic perspective, by framing opponent shaping meta-learning problem. In doing so, it decouples the task of interacting with the environment from that of influenc- ing the opponent's learning dynamies. This decoupling is achieved via bi-level agent architecture: an inner agent that interacts with its co-ple co-players, and an outer agent that updates or conditions the inner agent's policy. The outer agent operates in a meta-game, where the meta-state consists of all players' parameters, and the meta-action determines the inner agent's policy. Between episodes, other players update their parameters using their respective learning algorithms. This formulation enables the meta-agent to optimize for long- terg-term opponent shaping effects.
While M-FOS has demonstrated strong empirical results, including outperforming LOLA-based agents in the IPD, it presents scalability challenges due to its dual-agent architecture. To address these, Khan et a (2024) propose SHAPER, which simplifies M-FOS' architecture by collapsing the shaping agent into a single recurrent neural network (RNN). The key insight is the distinction between history and context within opponent shaping. History captures intra-episode information necessary for implementing conditional strategies such as TFT, while context captures inter-episode information about the opponent's learning dynamics. SHAPER captures history through the RNN's s inputs and context through its hidden state, which persists across episodes within trial. This unified architecture eliminates the dual action spaces of M-FOS, allowing the agent to operate directly in the environment's ' original action space. However, SHAPER is inherently tied to RNN architectures, with its mechanism relying on distinct memory streams for capturing history and context. Recently, Meulemans al. 20024) proposed COALA-PG, which also employs RNN-based sequence policies in the same meta- meta-learning setting but uses policy gradient algorithms instead of evolution strategies.
3 METHODOLOGGY
3.1 PRELIMINARIES
Agents interact with the environment by generating text. Let V denote the model's vocabulary, and W1.1 (wi...., wy),, with wt E V, vI E (1....,L) L). At each interaction, the agent's action is sequence of tokens sampled from:
L
Po oo c) = II
polur c, wei),
(1)
1=1
3

---

Published as a conference paper at ICLR 2026
where w-1 := w1--11 the context is the environment's description, and L is the generation length.
For simplicity. we set L = and define the distribution pa (w c), with w E V. as our agent's s policy. We deliberately avoid constrained decoding (Beurer- Kellner al. 2024 Ugare et al. 2024)
or format-specific fine-tuning, relying instead on textual instructions to guide the model toward the desired format.
We formalize our environments as repeated normal-form games (Fudenberg & Tirole, 1991]. I. Let M (I, (Ar (er) denote base game, where / - [1,...n) represents the of players. Each player i E has an associated action space A; and reward function R& Ai X .... X A.. + I. repeated formal-form game consists of playing M for T rounds, where T can be finite infinite. In this paper, we focus on the finite case. In each round t, players simultancously choose actions of for E The resulting joint action a' = (af,... a²,)) determines the reward rf = R; (a') that each player receives. The actions of are sampled from player-specific policies po, po, (w f(h")), where
fh²) is any function of the joint action history h' (al, ,..,,",,a"" a'-1) (e.g., the previous joint action 1))
3.2 OPPONENT SHAPING IN LLM AGENTS
We introduce ShapelLLM, a model-free opponent shaping algorithm designed to leverage the natu- ral language capabilities of LLMs.. Shapel.LM condenses both history and context into structured natural language prompts, explicitly capturing the two forms of memory required for shaping into one information stream. As in existing model-free algorithms (Lu al. 2022; Khan et al. 2024), interactions are organized into trials. Each trial comprises N parallel environments, where agents engage E episodes, each comprising T rounds of the specified matrix game (Figure provides schematic representation of a trial).
2
n
n ==
n=N
T rounds
T rounds
1=2
t=T
Figure 1: Schematic representation of trial. Each box corresponds an episode (a game played for T rounds). Same-colored boxes represent episodes within the same parallel environment. Within each environment, episodes occur sequentially as indicated by the arrows. The shaper updates its parameters using the experience collected throughout the entire trial.
Let E {1...., T) denote the round within an episode, and E (1....,)) denote the episode within trial. For notational simplicity, collapse the pair (e, into single timescale index x T). We formalize the shaping task as a POMDP (S, A, P. R. Ω, O,7). The state 3* = (0) c hher Š encodes the parameters and conditioning prompts of all LLM agents from the previous step. The action space A and reward function R. are equivalent those of the underlying repeated normal-form game. The observation o" = f(a! a² "?..,,"-1)) E Ω is function of the joint actions across all past steps. Lastly, P. and θ denote the state transition and observation functions, respectively, and 7 denotes the discount factor.
T E {1.....
At the beginning of training, players are initialized with policies (ee and receive initial ob- servations {ce hez specifying the game characteristics (number of players, action space, and re- ward matrix) and action labels. At the 7-th round of a trial, the shaper receives an observation cj that concatenates components: the most recent joint action (1- and compressed natural
4

---

Published as a conference paper at ICLR 2026
language representation of all the previous joint actions in the trial (f(a', ..... a)- This sep- aration captures the distinction between history and context. The shaper then samples an action aj Po, (w c.) and receives the corresponding reward rj and next observation c/+
The opponent update their policy parameters at the end of cach episode to maximize their return = rf. where rf is the reward obtained in round by player i from the game's payoff matrix (Appendix A.I). Consequently, within each trial, the shaper is exposed E opponent updates, though only indirectly through the evolving summaries of joint actions that persist across episodes. worth noting that opponent parameters are not reset between trials. By contrast, the shaper's own parameters
f are updated only at trial finalization to maximize the cumulative trial return J - J === E
((c.)) = Er=1 rj. where (e.t) denotes the shaper's reward in round of episode e. Crucially, both agents receive rewards from the same payoff matrix. While
present this formulation in the context of repeated normal-form games, the ShapeLLLM framework
generalizes to any environment that can be formulated partially observable stochastic game.
EXPERIMENTAL SETTINGS
4.1 ENVIRONMENTS
We investigate opponent shaping on iterated versions of four canonical x games. These environ- ments were selected to represent diverse incentive structures actoss strategic interactions.
Iterated Prisoner's Dilemma (IPD). Players choose between cooperation (C) and defection (D). Mutual cooperation yields the highest collective payoff, but each player faces individual incentives to defect and exploit cooperative opponents (Rapoport. 1974 Axelrod & Hamilton, 1981).
Iterated Matching Pennies (IMP). A zero-sum game where players choose between heads (H) and tails (T). One player receives a positive payoff when actions match, whereas the other rewarded when they differ. This environment is purely adversarial.
Iterated Chicken Game (ICG). Players can either Swerve (S) or Go straight (G). Going straight yields an advantage against swerving opponent, but mutual aggression results in catastrophic out- comes for both, creating a coordination problem under risk (Rapoport & Chammah) 1966).
Iterated Stag Hunt (ISH). Players choose between Stag (S) and Hare (H). Hunting stag yields the highest payoff but only if both players coordinate, while hunting hare offers a lower but guaranteed reward. This creates coordination problem with multiple equilibria.
We assign a single token Wa, to each action a, E A (c.g. "C" for cooperate and "D" for defect in the IPD), and treat any other generation Tnull, and the transition is excluded from both players' game histories (see Appendix A.2).
illegal action (Amul). Choosing amil incurs penalty
4.2 IMPLEMENTATION DETAILS
Our base model is gemma-2-2b-i (Gemma Team 2024), a small, Instruction-tuned, open-source language model. We focus on small models for computational efficiency and choose instruction- tuned variants as they are more goal-directed (Ouyang et al.] (202)) and benefit from coding data exposure (Duan al. 2024). To keep the agent architecture minimal, we restrict memory to the context window of the model and avoid additional reasoning scaffolds such as Mlain-of--hooueee (CoT) prompting (Wei al. 2022b), which are less effective in small models (Wei et al. 2022a).
We train our agents using QLoRA (Dettmers et al. (023). with the base model quantised to 4-bit precision via the BitsAndBytes package (Dettmers von Koeller 2022), and adapters of rank r = implemented through the PEFT library (Mangrulkar et al.] 2022). The learnable parameters comprise the LoRA adapters for the query/value projections and the value head parameters. All models are fine-tuned using a custom implementation of PPO that inherits from the TRL package]]
von Werra et al. 2020 We run PPO training for 200-300 trials, with N = parallel environments,
E 5 episodes, and T 20 rounds per episode. For the shapers, express context via cumulative
'The default implementation is only compatible with contextual handits. chub. "oode available wetttpp: t/g. com /wt.//mitttttttaameemmammm/maiia 11 1i/shapee11 1
5

---

Published as a conference paper at ICLR 2026
state visitation counts (e.g.. in the IPD: "CC: 1, CD: 1, DC: 2, DD: 3"). All training done a single A100 GPU with 40G of VRAM. The full specification of the hyperparameters, reward matrices, and training prompts used is provided in Appendices A.3 A.1 A.1I respectively.
5 SHAPING IN EXPLOITATIVE SETTINGS
We consider two training configurations for our agents across the IPD, IMP and ICG,
Baseline. We establish the bascline performance using two LLM-based naive learners (NL) that treat their opponent stationary component of the environment. Each agent's conditioning prompt contains the game description and the most recent joint action. Once the episode finished, both players simultaneously update their parameters via PPO maximize episodic returns. This baseline establishes expected behavior when no opponent shaping occurs.
Shaper vs. naive learner. An LLM-based shaper interacts with LLM naive learner (with the same configuration as in the baseline). As described in Section the shaper updates its parameters only at trial completion, after having observed multiple opponent parameter updates.
Following training, we evaluate performance by having each trained pair of agents play 100 games with the same episode length used during training (I" 20).
5.1 SHAAPING IN THE IPD, IMP AND ICG
We run experiments across 5 random seeds using action labels C, Was D for the IPD, Way H, Woz = T for the IMP, and Waj S, Waz for the ICG. Figure illustrates the training dynamics for the shaper experiments across the three games (the corresponding figures for the basclines are shown in Appendix AO.
Average Reward per step
IPD
IMP
ICG
0.5
2.5
Shaper Opponent
Shaper Opponent
Shaper Opponees
2-
0.0
2.0
-0.5
1.5
Mous
50
3.00
150
200
100
150
2.00
50
100
150
200
State Visitation
1.00
1.00
1.00
0.75
0.75
HHH
0.50
DD
TT
GG
0.25
0.25
0.00
50
100
150
200
50
1.00
150
200
50
100
150
200
Epoch
Epoch
Epoch
Figure 2: Average reward per step (top row) and state visitation (bottom row) during training for the shaping experiments across the IPD, IMP, and ICG. In the state visitation figures, the outcome "T" encompasses all transitions where either player chose auall- Results are reported along with a 95% confidence interval over 5 random seeds.
We represent the context via visitation counts instead of full trajectories prevent the token length from growing linearly with the number of rounds. For the ICG baseline, we conducted experiments over 10 seeds. Each seed converges to one of the two possible Nash Equilibria, a larger sample size is needed to accurately estimate the percentage of convergence each equilibrium.

---

Published as a conference paper at ICLR 2026
Table presents the post-training evaluation results, where each jointly-trained pair played 100 games with episode length = =00. The results demonstrate successful opponent shaping across all environments. For completeness, we evaluate performance across games of varying lengths (see Appendix A.7 and observe a similar performance.
Table I: Post-training evaluation results for the IPD, IMP, and ICG comparing baseline (two naive learner)) versus shaper-naive learner pairs. Average rewards per step are reported with 95% confi- dence intervals across 5 random seeds, except for the ICG baseline, where we use 10. Transitions involving Culll excluded (comprising 2% of actions in IPD, 0.1 in IMP, and 1% in ICG).
Baseline
One Shaper
Player
Player 2
Shaper
Opponent
IPD
1.00 + 0.00
1.00 0.00
3.96 + 0.01
0.10 0.04
IMP
-0.03 + 0.09
0.03 + 0.09
0.99 = 0.01
-0.99 + 0.01
ICG
2.00 + 0.58
2.00 0.58
2.98 = 0.01
1.01 = 0.01
We begin by examining the performance in the IPD. In the baseline, both learners converge to mutual defection, which is the Nash Equilibrium, achieving an average payoff of In contrast, the shaper achieves an average reward of 3.96, exceeding what any zero-determinant extortion (Press & Dyson, 2012] or tit-for-tat strategy could obtain. Meanwhile, the opponent achieves 0.1, which is lower than the mutual defection payoff. The training dynamies show three-phase -hree-pase pattern: starting from high initial cooperation, the shaper first sharply reduces its cooperation rate, then plateaus at a stable level to maintain opponent cooperation, and finally slowly decreases cooperation to achieve near-maximal exploitation.
We observe similar performance patterns in the ICG. The baseline results show average rewards of 2 for both players. The standard deviations of 0.58 reflect the fact that each seed converges to one of the two pure Nash equilibria: either (Swerve, Go Straight) (Go Straight, Swerve). In contrast, the shaper consistently achieves an average reward of 2.98 while limiting its opponent to 1.01. The training dynamics differ from the IPD: the shaper adopts an aggressive strategy by sharply reducing its swerving probability early in training, forcing convergence to its preferred equilibrium.
Finally, examining the IMP results, we observe that both agents in the baseline oscillate around the mixed Nash equilibrium with near-zero average payoffs (-0.03 and 0.03 respectively). In contrast, when one agent shaper, we observe clear exploitation, with the shaper obtaining a reward of 0.99 while the opponent obtains -0.99. The state visitation converges to equal frequency for the two states that favor the shaper: (H, H) and (T, T).
Collectively, these results demonstrate that the shaper consistently outperforms its opponent across all games, providing strong evidence of its ability to successfully influence opponents' learning dy- namics in adversarial and mixed-motive scenarios. To establish the robustness of these results, we conduct several additional experiments. First, we conduct an ablation study to determine whether shaping effects stem solely from the shaper's 's enriched observation space (Appendix A.4). We adopt the same settings as in the baseline experiments, with one of the agents receiving a summary of all interactions in the current episode. Second, we examine sensitivity to prompt variations by running shaping experiments with actions presented in reversed order and an alternative prompt formula- tion (Appendix A.5). These experiments confirm that ShapeLLM achieves robust shaping across different configurations, and that enriched observations alone are insufficient for effective oppo- nent shaping. Third, gain better insights into the shaping mechanism, we provide an ablation study in Appendix A.& in which we remove both the intra- and inter-episode history, and the inter- episode history only, from the shaper's observations in the IPD. These confirm that both intra- and inter-episode information are essential for effective shaping. Finally, to assess cross-model gener- alization, we explore the baseline and shaper experiments for the IPD using Llama-3.2- 1--211B--nnttrutt Llama Team, Al @ Metal 2024b) base model in Appendix A.9
5.2 ROBUSTNESS AGAINST DIFFERENT OPPONENTS
A robust shaping procedure should be capable of successfully influencing the learning dynamics of a diverse set of opponents. To test this capability, explore shaping against opponents with distinct
7

---

Published as a conference paper at ICLR 2026
initial policies. For each game, we systematically select three action label pairs yielding initial probabilities of playing action a fl of 0.75, 0.5, and 0.25. We then conduct shaping experiments using these labels for the opponents. A detailed description of the selection method, action labels, and initial output probabilities for each opponent is provided in Appendix [A.6] We train shapers against each selected opponent using the same procedure as in Section 5.1 with evaluation results shown in Table 2
Table 2: Post-training evaluation results for the shaping experiments in the IPD, IMP and ICG across different opponent initializations. Each column represents a distinct opponent characterized by its approximate initial probability of playing action 41- Average rewards per step are reported with 95% confidence intervals across 5 seeds. Transitions where Onuli is played by either player are excluded from the analysis (comprising 0-2% of all transitions).
PaL (a1)
0.75
P&L
0.50
(a1) 0.25
Shaper
Opponent
Shaper
Opponent
Shaper
Opponent
IPD
3.99 = 0.01
0.01 = 0.02
3.95 3.95:000 0.01
0.04 + 0.03
3.98 + 0.02
0.07 + 0.07
IMP
0.96 = 0.02
-0.96 + 0.02
0.99 0.01
-0.99 0.01
0.99 = 0.01
-0.99 + 0.01
ICG
3.00 = 0.00
1.00 = 0.00
2.99 + 0.01
1.01 + 0.01
2.95 = 0.01
1.05 + 0.01
Across all games and opponent types, shapers successfully exploit their co-players, achieving av- erage per-step rewards of 3.97 in the IPD, 0.98 in the IMP, and 2.98 in the ICG. In the IPD and ICG, opponents converge to less favorable outcomes when initialized with more cooperative poli- cies: their average rewards per step range from 0.01 to 0.07 in the IPD and from 1.00 to 1.05 in the ICG as initial policies become increasingly defective. The training dynamics (Appendix (A. reveal that shapers respond strategically to opponent initialization. Against more cooperative oppo- nents, shapers reduce their cooperation more rapidly and reach lower final cooperation levels. This pattern indicates that initially defective agents require more prolonged cooperation incentives before they can be effectively exploited.
In contrast, the IMP shows no sensitivity to different opponent initializations. Intuitively, shaping should be more challenging against initial policies closer to the mixed Nash equilibrium. However, our results reflect no such effect. is worth noting that the opponent with P&r ~ 0.5 is not initialized with a purely random policy, but with the closest approximation achievable through action label selection (see Appendix A. the exact initial policy). Shapers converge to near-optimal outcomes, achieving rewards of 0.96, 0.99, and 0.99.
SHAPING IN COOPERATIVE SETTINGS
In the ISH baselines both agents achieve a mean reward of 1.30, where 90% of runs converge to the Pareto-inferior equilibrium (both hunt Hare 7 With a shaper present, runs consistently converge to the Pareto-optimal equilibrium (both hunt Stag), both achieving rewards of approximately 3.96. This shows that shaping can resolve coordination failures and guide systems toward mutually beneficial outcomes when cooperation is required. In the cooperative IPD variant, baseline runs converge to the Nash equilibrium (mutual defection) with rewards of 1 each. With shaper, all runs achieve mutual cooperation, yielding rewards of 5.88 and 2.86 for the shaper and naive learner, respectively. This outcome demonstrates that shaping can achieve globally beneficial outcomes in environments where other players have mixed incentives.
'The action 0n corresponds to cooperation in the IPD, playing heads in the IMP, and swerving in the ICG. 'We considered a variant where shaper received sum both players' payoff as a reward. This configuration shifted the Nash equilibrium an asymmetric outcome where the shaper cooperates while the opponent defects.
We run the ISH and C-IPD baseline experiments across 10 seeds.
8

---

Published as a conference paper at ICLR 2026
We investigate whether shaping can guide in- teractions toward mutually beneficial rather than purely exploitative outcomes. We ex- plore this in two environments: cooper- ative variation of the IPD (C-IPD) and the
Average Reward per slep
C-IPD
ISH
Iterated Stag Hunt (ISH). For the C-IPD, we provide both agents with the original payoff matrix, but modify the shaper's ver-
Stape: Opp-onend
Stager Oppeones
sion so that its highest payoff is achieved through mutual cooperation, with all other payoffs unchanged This could be inter- preted the shaper receiving an intrinsic re- ward when the most globally beneficial out- come achieved. The reward matrices used can be found in Appendix A.I We use ac- tion labels Wa, = C, Wa, = D for the C- IPD, and Wa, S, Waz H for the ISH. Figure iillustrates the training dynamics for the shaper experiments across the two envi- ronments (the corresponding figures for the baselines are shown in Appendix I0. Ta- presents the post-tren training evaluation re- sults.
State Vislution
0:75
58
2.50
DC
0.20
155
DD
11:1
a.m
0.00
Epoch
Epoch
Figure 3: Average reward per step (top row) and state visitation (bottom row) during training for the shaping experiments across the C-IPD and ISH. In the state visitation figures, the outcome "I" encompasses all transitions where either player chose anull- All results are reported along with 95% confidence interval over 5 random seeds.
Table 3: Post-training evaluation results for the C-IPD and ISH comparing baseline versus shaper- naive learner pairs. Average rewards per step are reported with 95% confidence intervals across and 10 random seeds for shaping and baseline experiments, respectively. Transitions with Gnall are excluded from the analysis (~~ 2% and 0.1% of actions, respectively).
Baseline
One Shaper
Player
Player
Shaper
Opponent
C-IPD
1.00 + 0.00
1.00 + 0.00
5.88 + 0.03
2.86 0.02
ISH
1.30 + 0.52
1.30 + 0.52
3.96 + 0.02
3.96 + 0.02
7 DISCUSSION
Implications. In this work, we have demonstrated that LLM-based agents can be susceptible to opponent shaping in both exploitative and cooperative game-theoretic settings. As LLM agents become increasingly deployed in real-world applications, they will inevitably interact with other agents, and potentially train on data acquired through these interactions (for example, in the case of continually learning L.Ms).. In such settings, our findings suggest that agents could be vulnerable strategic exploitation by opponents with no knowledge or control over them. Conversely, the same mechanisms could be leveraged beneficially, enabling agents to guide interactions toward mutually beneficial outcomes regardless of the goals of their opponents.
Limitations. Our work has several limitations that suggest promising directions for future research. First, due to computational constraints, we have only evaluated our approach using a single small model (gemma-2-22) 2-2b-it). Future work could investigate whether shaping capabilities generalize to larger models and explore the relationship between model scale and shaping dynamics. For in- stance, whether smaller models are more vulnerable to these influences or whether larger mod- els possess enhanced shaping capabilities. Second, our experiments focus solely on interactions among LLM agents. Future work could investigate cross-architecture shaping dynamics, examining whether LLM-based shapers can influence the learning dynamics of other types of agents or vice versa. Third, our experiments instructed agents to select from a fixed set of action tokens. While this restriction made evaluation tractable, it limits the ways in which LLMs In practice, LLM agents can communicate through natural language, and expanding interactions be- yond fixed tokens may substantially alter shaping dynamics. Even within the same same-theoretic
influence
another.

---

Published as a conference paper at ICLR 2026
settings, agents could employ language strategically, for example, by signaling intentions negoti- ating before making a move. Future work could examine whether such natural language interaction strengthens, weakens, or qualitatively changes shaping outcomes. Finally, study was restricted to 2 x matrix games, where incentives are unambiguous and easily interpreted. Many real-world interactions, however, involve more nuanced overlapping objectives, where cooperation and com- petition are not strictly binary. Exploring shaping in environments with richer payoff' structures or multiple objectives would yield a deeper understanding of how these dynamics generalize more realistic settings.
CONCLUSION
In this paper, we have investigated whether opponent shaping, a well-establiseed technique in multi- agent reinforcement learning, extends to LLM-based agents. To the best of our knowledge, this is the first work to study opponent shaping with LLM agents. We proposed ShapeLLM, a model-free op- ponent shaping method for transformer-basee agents, and demonstrated successful shaping in both exploitative and cooperative settings. In exploitative scenarios, LLM shapers influenced opponent learning in repeated games such the IPD, IMP, and ICG, steering convergence toward outcomes that maximized their own payoff. In cooperative scenarios, shaping promoted coordination in set- tings like the ISH and modified IPD, guiding agents toward mutually beneficial equilibria. By demonstrating that LLMs can both shape and be shaped through interaction alone, our findings highlights the importance of understanding multi-agent dynamies when deploying these systems in shared environments.
ACKNOWLEDGMENTS
This work was supported by the UK Engineering and Physical Sciences Research Council (EPSRC) through the Centre for Doctoral Training Studentship in Cybersecurity EP/S022503// (Marta Emili Garcia Segura) and grant EP/X028569/1.
REFERENCES
Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. Nature Human Behaviour, 2025.
Anthropic. Claude Opus 4 Release.
2025a.
Anthropic. Claude 3.7 Sonnet and Claude Code. htttpe//wwwwanhhrpplc...omm
3- 2025b.
Robert Axelrod and William D. Hamilton. The evolution of cooperation. Science, 139 1396, 1981.
Jan Balaguer, Raphael Koster, Christopher Summerfield, and Andrea Tacchetti. The good shepherd: An oracle agent for mechanism design. arXiv preprint arXiv:2202. 101335,, 2022.
Luca Beurer-Kellnet, Mare Fischer, and Martin Vechev. Guiding LLMs the right way: fast, non- invasive constrained generation. In Proceedings of the 41 International Conference on Machine Learning (ICML'24), 2024.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS 20), 2020.
10

---

Published as a conference paper at ICLR 2026
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Eee Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro. and Yi Zhang. Sparks of artificial general intelligence: early experiments with GPT-4, arXiv preprint arXiv:2303. /2712, 2023.
Lucian Busoniu, Robert Babuska, and Bart De Schutter, comprehensive survey of multiagent rein- forcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C' (Applications and Reviews), 38(2):156-17,, 2008
Tim Dettmers and Titus von Koeller Bitse.dBytes. 2022.
https://githun com/
bitsandbytes- hotas/eeooooccttttotoooaaaaaaaa htt.b/iiaaaanbbybeseeeeebeee
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zeulemoyer. QLoRA:
Efficient Finetun-
ing of Quantized LLMs. In Proceedings of the 37th International Conference on Neural Informa- tion Processing Systems (NeurIPS'23) 2023.
Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel- Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. GTBench: Uncovering the strategic reason- ing capabilities of LLMs via game-theome theneeth evaluations. In Proceedings the 38th International Conference on Neural Information Processing Systems (NeurIPS 24), 2024.
Juan Agustin Duque, Milad Aghajohari, Tim Cooijmans, Razvan Ciuca, Tianyu Zhang, Gauthier Gidel, and Aaron Courville. Advantage alignment algorithms. In Proceedings of the 13th Inter- national Conference on Learning Representations (ICIRR255) 25), 2025.
Caoyun Fan, Jindou Chen, Yaohui Jin, and Hao He. Can large language models serve rational players in game theory? a systematic analysis. In Proceedings of the 38th AAAI Conference on Artificial Intelligence (AAAI'24), 2024.
Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent- awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS'I80) 2018.
Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Friederike Niedt- ner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, et al. Magentic-one: A gener- alist multi-agent system for solving complex tasks. arXiv preprint arXiv:2411.04468, 2024.
Drew Fudenberg and Jean Tirole. Game Theory. MIT Press, 1991.
Kanishk Gandhi, Dorsa Sadigh, and Noah D Goodman. Strategic reasoning with language models. arXiv preprint arXiv:2305.1 13066 2023.
Gemma Team. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.000 18, 2024.
Alireza Ghafarollahi and Markus J Buchler. SciAgents: automating scientific discovery through bioinspired multi-agent intelligent graph reasoning. Advanced Materials, 37(2))241525, 2025.
Mare Harper, Vincent Knight, Martin Jones, Georgios Koutsovoulos, Nikoleta E Glynatsi, and Owen Campbell. Reinforcement learning produces dominant strategies for the iterated prisoner's dilemma. PLOS One, 12(12):e01 88046, 2017
Jen-tse Huang, Eric John Li, Man Ho Lam, Tian Liang, Wenxuan Wang, Youliang Yuan, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, and Michael R Lyu. How far are we on the decision-making of LLMs? Evaluating LLMss gaming ability multi-agent environments. In Proceedings of the 13th International Conference on Learning Representations (ICLR") (25), 2025.
Akbir Khan, Timon Willi, Newton Kwan, Andrea Tacchetti, Chris Lu, Edward Grefenstette, Tim Rocktäschel, and Jakob N. Foerster. Scaling opponent shaping to high dimensional games. In Proceedings of the 23rd International Conference on Autonomous Agents and Multi-Agent Sys- tems (AAMAS' 24), 2024.
11

---

Published as a conference paper at ICLR 2026
Alistair Letcher, Jakob Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson. Stable opponent shaping in differentiable games. In Proceedings of the 7th International Conference on Learning Representations (ICLR') 19), 2019.
Llama Team, Meta. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783, 2024a.
Llama Team, A Meta. sion with
Llama 3.2: Revolutionizing edge AI and vi-
https://ail www.. ccm/blog/ 1lama 3- 2- connect- 2024-vision edge mobi devicess 2024b.
open, customizable models.
Christopher Lu, Timon Willi, Christian A Schroeder De Witt, and Jakob Foerster. Model-free opponent shaping. In Proceedings of the 39th International Conference on Machine Learning (ICML'22), 2022.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. PEFT: State-of-the-art Parameter-Efticien Fine-Tuning methods, https: com/ conn/huggingface/peftt 2022.
Alexander Meulemans, Seijin Kobayashi, Johannes von Oswald, Nino Scherrer, Eric Elmoznino, Blake Richards, Guillaume Lajoie, João Sacramento, et al. Multi-agent cooperation through learning-aware policy gradients. arXiv preprint arXiv:2410. 186366 2024.
Mistral-AL. Magistral. arXiv preprint wrxiv:2506.10111 2025.
OpenAI. Computer-using agent: Introducing a universal interface for Al to interact with the digital 2025a.
world. http:://ppapaa..cooooooe/////pps
OpenAl. GPT-5 Announcement,
ctt.sltooopppppp/p 2025b.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS 22), 2022.
Bo Pan, Jiaying Lu, Ke Wang, Li Zheng, Zhen Wen, Yingchaojie Feng, Minfeng Zhu, and Wei Chen. AgentCoord: Visually exploring coordination strategy for M.b.eb multi-agent collaboration. Computers & Graphics, 132: 102:104338, 2025.
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceed- ings of the 36th Annual ACM Symposium on User Interface Software and Technology (3)ST'23,, 2023.
Shishir Patil, Tianjun Zhang, Xin Wang, and Joseph Gonzalez. Gorilla: Large language model connected with massive APIs. In Proceedings of the 38th International Conference on Neural Information Processing Systems (NeurIPS 2024.
Giorgio Piatti, Zhijing Jin, Max Kleiman-Weiner, Bernhard Schölkopf, Mrinmaya Sachan, and Rada Mihalcea. Cooperate collapse: Emergence of sustainable cooperation in society of LLM agents. Advances in Neural Information Processing Systems (NeurIPS'24 24), 2024.
William H Press and Freeman J Dyson. Iterated prisoner's dilemma contains strategies that dominate any evolutionary opponent. Proceedings of the National Academy of Sciences, 109(26):10409- 10413, 2012.
Qwen Team. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Proceedings of the 38th International Conference on Neural Information Processing Systems (NeurIPS 24), 2024.
12

---

Published as a conference paper at ICLR 2026
Anatol Rapoport. Prisoner's dilemma recollections and observations. In Game Theory as Theory of Conffict Resolution, pp. 17-34. Springer, 1974.
Anatol Rapoport and Albert M. Chammah. The game of chicken. American Behavioral Scientist, 10(3):10-28 1966.
J Rosser and Jakob Nicolaus Foerster. AgentBreeder: mitigating the Al safety impact of multi-agent scaffolds via self-improvement. Proceedings the 13th International Conference on Learning Representations (ICLR 25), 2025.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS'23) 233), 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (Tdd::/70.0637777 2017.
Lior Shani, Aviv Rosenberg. Asaf Cassel, Oran Lang, Daniele Calandriello, Avital Zipori, Hila Noga, Orgad Keller, Bilal Piot, Idan Szpektor, Avinatan Hassidim, Yossi Matias, and Rémi Munos. Multi-turn reinforcement learning from preference human feedback. In Proceedings of the 38th International Conference Neural Information Processing Systems (NeurIPS'24) 24), 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and Daya Guo. Deepseekmath: Pushing the limits of 033000 2024.
mathematical reasoning in open language models. arXiv preprint
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Rad- ford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS 20), 2020.
Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas Griffiths. Cognitive architectures for language agents. Transactions on Machine Learning Research, 2023.
Elizaveta Tennant, Stephen Hailes, and Mirco Musolesi. Moral alignment for LLM agents. In Proceedings of the 13th International Conference on Learming Representations (ICLR' 25), 2025.
Shubham Ugare, Tarun Suresh, Hangoo Kang. Sasa Misailovic, and Gagandeep Singh. SynCode: LLM generation with grammar augmentation. Transactions on Machine Learning Research, 2024.
Alexander Sasha Vezhnevets, John P Agapiou, Avia Aharon, Ron Ziv, Jayd Matyas, Edgar A Auéñez-Guzmann William A Cunningham, Simon Osindero, Danny Karmon, and Joel Z Leibo. Generative agent-based modeling with actions grounded in physical, social, digital space using concordia. arXiv preprint trXiv:23/2.03666 2023.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. TRL: Transformer Reinforcement Learning. https://
2020.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. A survey large language model based autonomous agents. Frontiers of Computer Science, 18(6), 2024.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsumori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. arXiv preprint 220x:2206.07688 2022a.
13

---

Published as a conference paper at ICLR 2026
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS'22) 2022b.
Timon Willi, Alistair Hp Letcher, Johannes Treutlein, and Jakob Foerster. COLA: consistent learn- ing with opponent-learning awareness. In Proceedings of the 39th International Conference on Machine Learning (ICMI.'22) 2022.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong. Ming Zhang, Jun- zhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang. Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wenjuan Qin, Yongyan Zheng. Xipeng Qiu, Xuanjing Huang, Qi Zhang, and Tao Gui. The rise and potential of large language model based agents: survey. Science China Information Sciences, 68( (8(12):121101, 2025.
Fengli Xu, Qianyue Hao. Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng. Chenyang Shao, Yuwei Yan, Qinglong Yang, Yiwen Song, Sijian Ren, Xinyuan Hu, Yu Li, Jie Feng, Chen Gao, and Yong Li. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint arXiv:2501.09666 2025.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReACT: Synergizing reasoning and acting in language models. In Proceedings of the 11th Inter- national Conference on Learning Representations (ICLR'23) 2023.
Siliang Zeng, Quan Wei, William Brown, Oana Frunza, Yuriy Nevmyvaka, and Mingyi Hong. Rein- forcing Multi-Turn Reasoning in LLM Agents via Turn-Level Credit Assignment. arXiv preprint arXiv:2505. 11821, 2025.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yinggian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXhv:2303.18823 2023.
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. ArCHer: Training lan- guage model agents via hierarchical multi-turn RL. In Proceedings of the 41st International Conference on Machine Learning (ICML'24) 24), 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.0855 2019.
A APPENDIX
A.I PAYOFF MATRICES AND ILLEGAL ACTION PENALTIES USED DURING TRAINING
In Section 5] we consider three environments: Iterated Prisoner's Dilemma (IPD), Iterated Matching Pennies (IMP), and the Iterated Chicken Game (ICG). Table shows their corresponding payoff matrices, where each cell contains tuple representing the payoffs of the row (first entry) and column (second entry) players. Actions are represented with the following labels: "C" for cooperate and "D" for defect in IPD, "H" for heads and "T" for tails in IMP, and "S" for swerve and "G" for go straight in ICG.
Table 4: Payoff matrices for the three environments considered in Section 5 to explore shaping in exploitative settings.
(a) IPD
(b)
IMP
(c) ICG
C
D
H
T
S
G
(3, 3)
(0, 4)
H
(1.-1)
(-1,1)
S
(2,2) (1,3)
D
(4, 0)
11
T
(-1,1) 1)
(1.-1)
G
(3, 1) (55
14

---

Published as a conference paper at ICLR 2026
opponent shaping promote globally beneficial outcomes in two games: Iterated Stag Hunt (ISH) and a cooperative IPD variant (C-IPD). In the latter, one player receives an enhanced reward for mutual cooperation while all other payoff's remain unchanged from the standard IPD. Table [ shows the corresponding payoff matrices, where C-IPD uses the same action labels as IPD, and ISH uses "S" for stag and "H" for hare.
In Section 6
Table 5: Payoff matrices for two environments considered in Section 6 investigate shaping in cooperative settings.
(a) C-IPD
(b) ISH S
C
D
H
(6, 3)
(0, 4)
(4, 4)
(0, 3)
D
(4, 0)
1)
H
(3,0)
(I,
The penalty for generating an illegal action, Tmull, is always set to one unit below the lowest reward in each game's payoff matrix. Specifically; - ISS 11
nall
1,
IMP null
-2, ,
-6 and
A.2 LLM GAMEPLAY IN REPEATED NORMAL-FORM GAMES
All the environments considered are 2x2 repeated normal-form games, Since our agents are LLM- based, even after restricting the generation length to one token, their output space much larger than the game's s action space ((V) >> |A| 2).
To tackle this space mismatch, one could try shrinking the model's 's output space via logit masking or rejection sampling. However, these interventions can alter the masked logits in unexpected ways or lead to increased computational time. Instead of actively ignoring or hiding parts of the output space, we define mapping Φ V A.
Directly defining such a mapping would require distributing the entire vocabulary across two actions, resulting in semantically unrelated tokens being mapped to the same action. This would impose an unnecessary learning objective whereby agents must learn arbitrary semantic equivalences that are orthogonal to the underlying strategic objective. To avoid this, we introduce null action Guulls such that AC = A, U }- This action is not meant to represent refusal to engage in the game, but rather failure to produce reasonable answer.
We then define Φ; : V + Ai. This formulation general and can accommodate open-ended generation if the mapping itself is another language model. For simplicity, choose:
a1
if Way
(.(w)=
a2 aguil
West
otherwise
such that each action can be played with one specific token, while any other token is considered illegal. The generation is steered towards this format via textual instructions (e.g., Reply with "C" or "D").
Augmenting the action space requires extending the payoff matrix. Table shows the augmented payoff matrix for a general 2x2 matrix game. The revised matrix is identical to the original one when both players play legal actions. If an agent plays amall, it receives a penalty Tmall regardless of its opponent's 's move. the agent plays a legal action but its opponent does not, the transition discarded.
With the augmented action space and payoff structure defined, we can now describe how agents interact within this framework. In the t-th round, each agent receives a context cf, which is sequence tokens consisting of the game description and a summary of the previous rounds of the game. A single token is then sampled from the output distribution of each agent (wf po, (wlc))) and subsequently mapped to a game action, such that a' = {af = dr(w)))ee The environment then returns the corresponding rewards ((rf = R,(a') (a((a)))e)) and contexts for the next round (fel+1 [cl, a²) ae)))))
15

---

Published as a conference paper at ICLR 2026
Table Augmented payoff matrix for training LLM-agents repeated normal-form games. When both agents play legal actions, payoffss match those of the underlying game. When an agent plays an illegal action, it receives a penalty Tmull, regardless of the opponent's action. However, when an agent plays a legal action but the opponent plays an illegal action, the transition is discarded as it provides no meaningful learning signal (indicated by dashes in the matrix).
a1
a2
Anlll
r(a1, r(a2, 01)
r r(a1; (2)
a2 agull
r r(a2, (2))
Tnull
Tnull
Tnull
A.3 TRAINING IMPLEMENTATION DETAILS FOR REPRODUCIBILITY
This section provides detailed hyperparameter specifications that supplement the implementation
details in Section 4.2
Generation Parameters. We use the same generation parameters for all agents across the training and evaluation phases. The configuration is kept to the default values with three exceptions: we en- able sampling ado_sample-ere True), disable top-k generation (top_k-0) and restrict the genera- tion length to one token www.x/eettokkee. top-p=1. 0) remain at default values.
1). All other parameters (e.g., temperature
Adapter Configuration. We employ the same adapter configuration for all agents. We only mod- ify the rank parameter (r=2) to reduce compute. All other parameters (c.g., lora alpha=32, lora dropout=0: 05, target modules ["q proj", "v proj"]) kept at their default values.
A.3.1 NAIVE LEARNER HYPERPARAMETERS
We aimed to maintain hyperparameters at their default values for naive learners to simulate realistic scenarios where opponents cannot be controlled. However, several adjustments to memory constraints and training instability. To reduce the memory consumption and compute, we reduced the adapter rank, batch size, and mini-batch size. These parameters both shapers and naive learners to ensure fair comparison. Additionally, we observed that under the default settings some agents learned generate a substantial amount of illegal actions. To counter this instability, we reduced the learning rate, the number of optimization epochs per batch (PPO epochs), and incorporated reward scaling to maintain stable training dynamies across all agents.
necessary due
kept identical for
Table 7 presents the hyperparameters used for training naive learners across all experiments rameters marked with the "(default)" flag indicate values that remained unchanged from the TRL library's ' default configuration.
N Pa-
A.3.2 SHAPER HYPERPARAMETERS
For shapers, the hyperparameters used varied across games and opponents. However, several core parameters were held constant across all experiments, which are presented in Table g Where pa- rameters deviate from default values, modifications were made either for computational efficiency (rank, batch size, mini batch size) or improved training stability (score scaling, PPO epochs)
We varied three main hyperparameters across experiments: the learning rate (Ir), the value function coefficient (CyF) and the clipping range (Ep). The learning rate and clipping range were reduced mainly increase stability during training. Under the default settings, convergence outcomes across different random seeds. The value function coefficient Cvr weights the value function term in the PPO loss. Shapers operate requiring them to predict expected returns at the trial level rather than the episode level. This creates a challenging value prediction problem: the value function must estimate returns ranging from entire
observed high variation in
much longer horizons than naive learners,
'These parameters were used for all naive learners and games except IMP. Under these parameters, naive learners playing the IMP converged to deterministic strategies rather than the expected mixed Nash equilibrium. We therefore reduced the learning rate to 1.41 x 10- and the value function coefficient to 0.05 for all IMP naive learners..
16

---

Published as a conference paper at ICLR 2026
Table 7: Naive learner hyperparameters used in all experiments. The "(default)" flag indicates hyperparameters taking the default value in the TRL package.
Parameter
Value
LoRA rank
LoRA target modules
["q "qy-proj". "v "v-proj"] (default)
Learning rate
1.41 x 10-6
Use adaptive KL control
Yes (default)
Starting KL coefficient
0.2 (default)
Target KL value
6.0 (default)
Horizon for adaptive KI. control
10000 (default)
GAE y
1.0 (default)
GAE A
0.95 (default)
Clipping range
0.2 (default)
Value Function clipping
0.2 (default)
Value Function Loss coefficient
0.2 (default)
Batch Size
100
Mini Batch Size
10
Gradient Accumulation Steps
1 (default)
PPO epochs
Score normalization
No (default)
Score scaling
Yes
Table Shaper hyperparameters fixed across all experiments. The "(default)" flag indicates hyper- parameters taking the default value in the TRL package.
Parameter
Value
LoRA rank
LoRA target modules
["q-prroj", "v "v-proj""] (default) Yes (default)
Use adaptive KL control
Starting KL coefficient
0.2 (default)
Target KL value
6.0 (default)
Horizon for adaptive KL control
10000 (default)
GAE
1.0 (default)
GAE
0.95 (default)
Value Function clipping
0.2 (default)
Batch Size
100
Mini Batch Size
10
Gradient Accumulation Steps
(default)
PPO epochs
Score normalization
No (default)
Score scaling
Yes
trials (for initial states) to single immediate rewards (for final states). Since the shaper's value head randomly initialized, under the default ovF value, the initial value loss dominates the policy loss by orders of magnitude, leading to large gradients and causing training instability. A particularly problematic consequence is that agents sometimes learn to generate illegal tokens simply to make the value prediction task easier. We address this issue by reducing the value function coefficient to better balance the relative contributions of the value and policy losses.
Table shows the learning rate, value function coefficient, and clipping range used to train the shapers across all experiments in the main text. The remaining hyperparameters used during training for all experiments shown in Table
17

---

Published as a conference paper at ICLR 2026
Table Shaper's learning rate (Ir), value function coefficient (cvp), and clipping range (Ep) for the experiments in Sections 5.1
5.2
Experiment
Ir
CYF
'p
IPD
1.41 X 10-7
10-3
10-4
Section 5.1
IMP
3,41 10-7
10-3
2 10
ICG
1.41 X 10-7
10-3
2 x 10
IPD with p (a)) 0.75
1.41 10-7
10-3
2 10
IPD with PNr (a1) 0.5
1.41 X 10-7
5 10
6x10
IPD with o ((1) ~ 0.25
1.41 X 10-7
30-3
10-4
IMP with PNr. (a)) 0.75
4,41 X 10-7
10-3
2x 10
Section 5.2
IMP with (a1) 0.5
4.41 X 10-7
10-3
2 2x 10
IMP with PNi. (a1) ~ 0.25
6.41 X 10-7
10-3
2 10-1
ICG with PNL (a1) ~ 0.75
1.41 X 10-7
10-3
2 10-1
ICG with PNt (a1) 0.5
1.41 10-7
10-3
2 10-1
ICG with PNI. (a1) ~ 0.25
6.41 X 10
10-3
2 X 10
C-IPD
8.41 10-8
5 10-5
2 10
Section
ISH
8.41 X 10-8
10-3
2 x 10-1
A.3.3 PACKAGE VERSIONS
We used the following versions for the main Python packages:
BitsAndBytes: v0.45.0
PEFT: v0.14.0
torch: v2.5.1
transformers: v4.47.0
TRL: v0.11.4
A.4 ABLATION STUDY: NAIVE LEARNERS WITH ENRICHED OBSERVATIONS
We conduct variation of the baseline experiments to test whether shaping effects we observe solely a product of the shaper's enriched observation space. The experimental setup involves two naive learners, with one player's 's observation space augmented to include the state counts from all previous interactions in the current episode. Crucially, these observations boundaries (unlike the shaper, whose observations persist across episodes within a trial). To ensure comparability, we use the same action labels in Section 5.1 Wai C, Waz for the IPD,
reset episode
Wa, H, was = T for the IMP, and Wa S, Waz hyperparameters presented in Table 7
G for the ICG. Both learners use the
Table 10: Post-training evaluation results for baselines where Player 1 receives enriched observa- tions. Columns ICG and ICG (alt. opp.) correspond to experiments with varying action labels for Player 2 (ICG uses Wa, = S, Wa, = G. and ICG (alt. opp.) employs Wa, = N, Wez = M). Average rewards per step reported with 95% confidence intervals across 5 random seeds.
IPD
IMP
ICG
ICG (alt. opp.)
Player (enriched) Player
1.00 + 0.00
-0.05 + 0.09
2.60 + 0.76
1.00 + 0.00
1.00 + 0.00
0.05 0.09
1.38 + 0.74
3.00 - 0.00
Table 10 reports the post-training average reward per step for the IPD, IMP, and ICG. In the IPD and IMP, despite the asymmetry in observations, outcomes mirror those in the standard baseline:
18

---

Published as a conference paper at ICLR 2026
mutual defection (1 each) for the IPD, and oscillation around the Nash Equilibrium (-0.05 and 0.05) for the IMP. This equivalence to the baseline is not the same for the ICG. When both players use action labels Wa, S, Way G (row ICG), the player with enriched observations achieves 2.60, while its opponent receives only 1.38. As in the baseline, each run converges to one of the two pure Nash equilibria, but 80% of runs favor the equilibrium most beneficial the player with enriched observations. At first glance, this suggests that the additional information, rather than training with ShapeLLM, could drive the shaping behavior observed in the ICG.
Average Reward per step
IPD
IMP
ICG
ICG (alt. opp.)
Player Player
Player
0.5
Player
2
2
Player Player 2
0.0
-0.5
Player Player 2
200
100
2000
100
200
1.00
1.00
1.00
State Visitation
0.75
CC
HI
0.75
SS
0.75
SS
TH
SG
SG
0.50
0.50
0.50
0.:00
0.25
0.25
0.25
0.00
0.00
0.00
0.00
100
200
200
100
200
100
200
Epoch
Epoch
Epoch
Epoch
Figure 4: Average reward per step (top row) and state visitation (bottom row) during training for the enriched observation baseline experiments across the IPD, IMP, and ICG. For the latter, two opponent configurations are presented: ICG and ICG (alt. opp.). They use Wa, S, Way G and Wa. = N, Wny = M as the opponent's action labels respectively, and Wa = S. was = G for the player with enriched observations. In the state visitation figures, the outcome encompasses all transitions where either player chose Gnull- The results are reported along with 95% confidence interval over 5 random seeds.
Inspecting the training dynamics (Figure 4. we observe a skewed initialization in the ICG, with 40% of transitions being (Go Straight, Swerve) training initiation. To test whether the obtained results were a consequence of this initialization, we conduct a control experiment where Player selects actions with labels Way N, Waz M" For this setup, the initial state visitation starkly different (see Figure 4p, with 46% of transitions being (Swerve, Go Straight). In this case, the enriched baseline for the ICG consistently converges to the least favorable equilibrium for Player (with rewards of 1.00, 3.00 for Players 1 and 2, respectively).
This control shows that the apparent shaping advantage in the ICG enriched baseline was driven by favorable initialization, not by the enriched observation space itself. When the initialization is modified, the enriched learner systematically converges to the least favorable outcome. In contrast, when ShapeL.LM is applied to the same opponent configuration, the shaper consistently achieves maximum rewards (see Section 5.2p. Taken together, these results demonstrate that enriched obser- vations alone are insufficient to produce shaping. Shaping requires the ability to indirectly observe and respond to the opponent's 's learning dynamics across episodes.
A.5 ROBUSTNESS TO PROMPT VARIATIONS IN EXPLOITATIVE SHAPING EXPERIMENTS
We conduct two experiments for robustness to prompt variations. The first variation uses prompt where the payoff matrix is presented in table form.
These labels used in Section 5.2 to show the shaping rohustness under different opponent initializations.
19

---

Published as a conference paper at ICLR 2026
Table-format hase prompt for the IPD with Way = C, Mlag
<bos><start of turnauser
You are playing a 2 2-player game with actiona: follows:
C, D. Points are assigned as
**D**
+**+* (3, 3) (0, 4) (1, 1) .* (4, 0)
Choose an action for the current reund. Reply only with or D. <end.of.turn>
<startt.of.turnmmoddll
Table-format prompt variation for the IPD. Instead of a textual description, the payoff' matrix is presented in markdown table form, replicating the base model's formatting style.
Figure 5:
Table 11: Post-training evaluation results for the IPD, IMP, and ICG with the table-format and switched-order prompts. Average rewards per step are reported with 95% confidence intervals across 5 random seeds. Transitions where anull is played by either player excluded from the analysis (comprising 0-4% of all transitions).
Table-format
Switched-order
Shaper
Opponent
Shaper
Opponent
IPD
3.50 + 0.16
0.53 + 0.09
3.99 + 0.01
0.02 + 0.01
IMP
0.94 + 0.06
-0.94 0.06
0.98 + 0.04
-0.98 0.04
ICG
2.99 + 0.01
1.01 + 0.01
2.77 = 0.37
1.23 + 0.37
To determine the specific table formatting, we ask the base model to generate general payoff matrix and replicate its output format (including spacing and header formatting). F shows the table-format base prompt used for the IPD.
For the IMP and ICG, the prompts are identical except for the modified action labels and payoff matrices. The example in Figure [ shows the base prompt that both players receive at training initiation. The dynamic updates of this prompt throughout training (i.e., how history and context updated) remain unchanged from those used in the main experiments (see Appendix(A.1l
We use the same action labels in Section 5.1 Way C, Way D for the IPD, Waj H, Way T for the IMP, and war = S, Was = G for the ICG. Figure 6 shows the training dynamics for the three games under the tabular prompt variation. initialization of the shaper and its opponent change significantly, especially for the IPD. Table shows the evaluation results obtained.
worth noting that with this new prompt, both the
The results demonstrate successful shaping across all three games. In the IPD, the shaper achieves 3.5 while its opponent receives 0.53, exceeding mutual cooperation payoffs but underperforming compared to the main text results. Since Section 5.2 shows consistent outcomes across different op- ponent initializations, this gap likely stems from the shaper's own initialization rather than opponent effects. The IMP shows a similar pattern with substantial initialization differences, with the shaper achieving 0.94 versus the opponent's 's the main setup, results are 2.99 for the shaper and 1.01 for the opponent.
-0.94. For the ICG, where initialization more closely matches
For the second prompt variation, we reverse the order in which actions are presented (see Figure As with the table format experiments, the base prompts for IMP and ICG are identical except for their respective action labels and payoff matrices. We use the same action labels as the previous variation.
20

---

Published as a conference paper at ICLR 2026
Average Reward per step
IPD
IMP
ICG
0.5
Shaper
Shaper
Shaper Opponees
0.0
2.0
Upponent
Opponent
-0.5
1.5
1.0
90
1.00
150
200
1.00
150
200
50
100
150
200
State Visitation
1.00
1.00
1.00
0.75
CC
0.75
EEE HT
5G
0.50
TII
DD
0.25
100
150
200
50
Epoch
Epoch
Epoch
Figure 6: Average reward per step (top row) and state visitation (bottom row) during training for the shaping experiments with the table-format prompt across the IPD, IMP, and ICG. In the state visita- tion figures, the outcome "I" encompasses all transitions where either player chose anull. Results are reported along with a 95% confidence interval over 5 random seeds.
Switched-order base prompt for the IPD with Waj = C, Wag D
boot>sstart.of.turmbbss
You are playing a 2 2-player game with actions: D, c. Points are assigned as follows: D/D: 1/1, D/C: 4/0, C/D: 0/4, c/c: 3/3. Choose an action <end.of.turn> <start.of.turn
current reund. Reply only with
Switched-order prompt variation for the IPD. The actions and payoffs are presented in reversed order compared to the main text prompt.
Figure 7:
Figure shows the training dynamics for the switched-ordee prompt experiments. As with the table- format variation, both agents exhibit substantially different initializations compared to the main experiments. Here, both players heavily biased toward playing action aı (cooperate in the IPD, heads in the IMP, and swerving in the ICG), with their policies initially being almost deterministic (average of 99% of (a1, 01) state trial initiation). This creates challenges for the IPD and ICG, where this joint outcome yields acceptable rewards for both players. Consequently, without suffi- cient exploration incentives, the agents' policies remain unchanged for some of the seeds, focusing exclusively on the value prediction problem.
To address this issue, we introduce an entropy regularization term to the PPO loss function. While this term was included in the original PPO formulation (Schulman et al. 2017), it is not implemented in the TRL, package. Since our agents' action space encompasses the entire vocabulary, maximiz- ing their output distribution entropy would create conflicting incentives: illegal actions would incur penalties while simultaneously reducing the loss through increased entropy. Instead, we extract logits only for the two allowed action tokens and compute the entropy of the resulting normalized distribution. To prevent convergence to suboptimal policies, we employ a decaying entropy coeffi- cient that gradually reduces the exploration incentive during training.
The wwitched-order prompt results are presented in Table IT With entropy regularization applied to both the IPD and ICG, again observe successful shaping with results closely matching those
21

---

Published as a conference paper at ICLR 2026
Average Reward per step
IPD
IMP
ICG
M
2.5
Shaper Opponent
Shaper Opponent
Shaper Opponeet
1.5
1.50
2.00
State Visitation
HH
TH
DD
TT
GG
0.25
100
150
200
100
Epoch
Epoch
Epoch
Figure 8: Average reward per step (top row) and state visitation (bottom row) during training for the shaping experiments with the switched-order prompt across the IPD, IMP, and ICG. In the state vis- itation figures, the outcome "T" encompasses all transitions where either player chose Gaull- Results are reported along with a 95% confidence interval over random seeds.
in the main text: 3.99 for IPD and 2.77 for ICG. In the ICG, while the shaper achieves an average reward of across most runs, for one of the seeds it consistently fails to explore alternative actions, converging to the outcome where both players swerve. Lastly, the IMP performs similarly other prompt formulations (0.94 for the shaper vs -0.94 for its opponent) without requiring an entropy regularization term.
These experiments demonstrate that ShapeLLM maintains robust shaping capabilities across differ- ent prompt formulations. Even when initial policies are nearly deterministic (as the switched- order variation), introducing entropy regularization to encourage early exploration enables effective opponent shaping. For reproducibility, we list the hyperparameters modified this appendix in Table 12
The rest of the hyperparameters used are specified in Table
Table 12: Shaper's learning rate (Ir), value function coefficient (evp), clipping range (ep), and entropy regularization parameters init cs Ts) for the experiments in Appendix A.5
Experiment
Ir
CvF
Ep
CS
init
cend
Ts
IPD
1.41 x 10-7
10-3
10 10-5
Table-format prompt
IMP
2.41 x 10-7
10-3
2 10
ICG
1.41 x 10-7
10-3
10-1
IPD
6,41 x 10-7
10-3
2 x 10
0.1
0.0
25
Switched-order prompt
IMP
6.41 x 10-6
10-3
2 10
ICG
1.41 x 10-7
10-3
2
10
0.7
0.0
25
A.6 ACTION LABEL SELECTION FOR ROBUSTNESS EXPERIMENTS
LLM agent output distributions vary significantly with the choice of action labels (Wa, Wa.). For the robustness experiments, we systematically selected action labels to achieve target initial output distributions for the opponents.
22

---

Published as a conference paper at ICLR 2026
Table 13: Action labels that produce the closest output distributions the target initial distributions for the IPD, IMP and ICG.
PÅL (a1)
~ 0.75
(a1) ~ 0.50
PNL (a1)
0.25
Wai
Waz
Wa.
Waz
Wai
Waz
IPD
H
K
N
Y
X
IMP
S
Y
N
Y
M
ICG
T
K
N
M
T
F
Target Distributions. For each game, we want to obtain opponents with initial probabilities of playing action
10 of approximately 0.75, 0.5, and 0.25.
We extract the LLM's output probabilities for these 5 prompts across all possible combinations of capital letters as action labels (325 total combinations). For each combination, we calculate the KL divergence between the target and extracted output distributions, averaged across the 5 prompts, and select the combination with the lowest divergence. The resulting action labels are shown in Table [3
Selection Procedure. Opponents encounter 5 different prompts during training I
The extracted initial probabilities for each pair of action labels in Table 13 are shown in Table 14 with separate results for the IPD a IMP (b). and ICG (c).
Table 14: Naive learner's initial action probabilities across three games (IPD, IMP, ICG) under varying action labels for the 5 distinct prompts encountered during training.
(a) Initial cooperation probability in the IPD.
Action Labels
p (C)
ph (C CC)
P&. (C CD)
PNI. (C) DC)
P DD)
Wal H, Waa K
0.60
0.89
0.89
0.70
0.68
Cal N, Wa2 Y
0.68
0.38
0.87
0.58
0.77
Wal = I, Waz X
0.12
0.21
0.75
0.34
0.24
(b) Initial probability of playing heads the IMP.
Action Labels
P&r. (H)
PQr. (H HH)
PRr. (H HT)
P&. [
TH)
Ph (H
TT)
Wal S, Wa2 Y
0.71
0.78
0.87
0.57
0.70
Wal = N. Wa2 = Y
0.48
0.32
0.60
0.56
0.66
Wal N, Woz M
0.30
0.20
0.36
0.13
0.18
(c) Initial swerving probability in the ICG.
Action Labels
PNL (S)
P&L(S (S SS)
PNL(S (S SG)
P&L (S
GS)
O (S GG)
Wa1 = T, Waz =
0.61
0.90
0.87
0.82
0.82
Wa1 N, Waz M
0.41
0.40
0.71
0.37
0.61
Wal = T, Waz = F
0.24
0.47
0.47
0.02
0.11
A.7 SHAPER EVALUATION FOR VARYING GAME LENGTHS FOR THE IPD, IMP, AND ICG
Table 15 presents the evaluation results for the shapers trained in Section 5.1 for varying game lengths (T 50, T 100). The results demonstrate that shaper performance remains consistent
The action a corresponds cooperation in the IPD, playing heads in the IMP, and swerving the chicken game.
11 One stateless prompt training initiation, and 4 prompts corresponding to the possible joint actions from the previous round g., the IPD: CC, CD, DC, DD).
23

---

Published as a conference paper at ICLR 2026
across different game lengths, with no significant degradation in exploitation capability as episodes become longer.
Table 15: Post-training evaluation results for the IPD, IMP, and ICG comparing baseline (two naive learners) versus shaper-naive learner pairs for varying game lengths (7 = 20, 50, 100). Average rewards per step reported with 95% confidence intervals across 5 random seeds. IIlegal actions are excluded from the analysis (comprising 2% of actions in IPD, 0.1% in IMP, and 1% in ICG),
IPD
IMP
ICG
Shaper
Opponent
Shaper
Opponent
Shaper
Opponent
T = 20
3.96 + 0.01
0.10 + 0.04
0.99 + 0.01
-0.99 + 0.01
2.98 0.01
1.01 + 0.01
50
3.97 + 0.01
0.09 + 0.04
0.99 + 0.01
-0.99 0.01
2.99 +.01
1.01 + 0.01
T = 100
3.97 0.01
0.09 = 0.04
0.99 + 0.01
-0.99 + 0.01
2.99
1.01 + 0.01
A.8 ABLATION STUDY: VARYING INTRA- AND INTER-EPISODE HISTORY IN THE IPD
We conduct two variations of our shaping experiments to test whether both intra- and inter-episode history are necessary for successful opponent shaping. Both experimental setups involve one shaper and one naive learner. In the first variation, the shaper receives only the current round's 's state, with no access to prior rounds or episodes. The shaper's 's parameters are still updated at the end of each trial, maintaining the asymmetric parameter update characteristic of Shapel.L.. In the second variation, we relax this constraint, with the shaper now receiving the full intra-episode history but no inter- episode information. In practice, this corresponds to the shaper's observations being reset at the end of each episode. Consequently, the shaper does not indirectly observe its opponent's parameter updates. The shaper's parameters are still updated only at the end of the trial. We test these variations in the IPD, with the hyperparameters used for the opponent and the shaper listed in Tables and 9 respectively.
Table 16: Post-training evaluation results for the IPD with one shaper against a naive learner, where the shaper receives varying levels of intra- and inter-episode history. Average rewards per step reported with 95% confidence intervals across 5 random seeds. Transitions with Guall are excluded from the analysis (~ 1% of actions).
Current state only
Intra-episode history only
Shaper
0.99 + 0.00
0.99 0.01
Opponent
1.02 + 0.01
1.02 + 0.02
Table the post-training average reward per step for the IPD when the shaper only receives the current state in its context, and when it receives intra-episode information only. In the first variation, remove all intra- and inter-episode history from the shaper's 's context test whether asymmetric parameter updates alone are sufficient for shaping to occur. In this setting, we obtain convergence to mutual defection, similarly to the baseline behavior, indicating that without addi- tional history information shaping cannot occur despite asymmetric updates. For the second varia- tion, in which intra-episode context is restored, we observe a similar pattern, with the confi guration converging to mutual defection. This demonstrates that inter-episode information is essential for shaping. such that the shaper able to indirectly observe its opponent's learning dynamics through context that persists across episodes.
Figure 9 shows the training dynamics for both ablation studies in the IPD. both cases, the asym- metric parameter updates influence the learning dynamics. Unlike the baseline, where both agents quickly converge to mutual defection, here the shaper maintains a transient advantage over its op- ponent during early training. This advantage is more prolonged when intra-episode information is included, with agents maintaining higher cooperation rates, likely due to the in in-context adaptation that additional history enables. However, the shaper ultimately fails to sustain opponent cooperation in both configurations, demonstrating that asymmetric parameter updates alone are insufficient to shape opponent learning dynamics.
24

---

Published as a conference paper at ICLR 2026
Average Reward per step
State Only
Intra-Episode Only
Shaper Oppponest
Shssper Oppoonent
State Visitation
OC
CC
DC
DD
DD
100
150
50
Epoch
Epoch
Figure 9: Average reward per step (top row) and state visitation (bottom row) during training for the IPD ablation experiments. The shaper receives either only the current state (left) full intra-episode history (right), with no inter-episode information in either case. In the state visitation figures, the outcome "T" encompasses all transitions where either player chose dmall. The results are reported along with 95% confidence interval over random seeds.
A.9 PRELIMINARY CROSS-MODEL VALIDATION SHAPELLM
ShapeLLM is designed to generalize across different LLM architectures, relying only natural language context without any architecture-specifie components. To provide preliminary evidence of cross-model generalization, report baseline and shaper experiments for the IPD using Llama- 3.2- 2--Ir-tnttr (Llama Team, AI@ Meta 2024b) as the base model.
The possible initial policies of Llama-3 3. 2-2-BB--sssree differ substantially from those of gemma- 2-2b-it. As discussed in Section 5.2 gemuma-2-2b-i'' initial policy for a given prompt depends on the choice of action labels. Figure T0 (a) shows heatmaps of gemmaa2-2b-it ''' initial output probabilities for tokens Wa, and Was across all 650 possible capital-letter action label pairs. Each heatmap corresponds to a different conditioning prompt (stateless or one of the four possible joint action states). Most policies concentrate in the diagonal (where p(w., (c) (c)+p((wglec) R 1), indicating a low probability of generating illegal actions. Additionally, despite the variation across action labels, policies for each conditioning prompt cluster in distinctive regions, indicating that the base model meaningfully differentiates between states.
In contrast, Figure 10 (b) shows that Llama-3.2-1 I-Intrruct produces substantially different behav- iors. Nearly all policies lie off-diagonal, indicating 10-20% illegal action rates for most action label pairs. Moreover, unlike gemma-22 2-2b-it. policies do not cluster distinctly by conditioning prompt. The heatmaps for different states exhibit similar distributional patterns, suggesting the model does not meaningfully differentiate between game states based on previous joint actions.
To address these challenges and isolate architectural effects from differences in initialization, we perform supervised fine-tuning (SFT) align gemma-2-2b-it. For the SFT procedure, we create a dataset of 1000 prompts that agents might encounter during IPD gameplay Each prompt has an associated target distribution, which cor-
initial policy with that of
Foror each prompt, uniformly sample episode number, round number, and previous joint action. For state-oceurrence prompts, additionally sample counts from uniform multinomial distribution.
25

---

Published as a conference paper at ICLR 2026
c= None
Way Wa
Way Wa
Way Wa.
Wa. Wa
p(Wa, [c)
(a)
0.3
0:0
0.23
0.25
0.25
0:25
600
0.00
0.00
0.00
8:00
0.5
1.00
1.00
1:00
3.00
plwa, [c]
0.55
0.5
0:75
(b)
0.00
1.0
10
1.00
plWai [c]
607
0075
0.5
0.15
8:7
(c)
6:50
91
0.90
4.9
100
0.59
8:10
923
0:2
p(Wa,()) (c)
plWa (c)
P[Wa/[))
p(Wa,[c)
p(Wa,Ic)
Figure 10: Initial policy distributions in the IPD for all possible two-letter action labels for (a) gemma--222-ie (b) 1aama-3.2-185 3.-ma-332--BIIsstrucc with default initialization, and (c) Llama-3.2- 2-2 1eae-3.2-/B-mniiie I-Insructt after SFT to match remmaa-2-2biit initialization. Each column corresponds to one of the five possible states encountered during IPD gameplay. For each state, extract the model's initial cooperation and defection probabilities across all possible two capital letter action label pairs. Each heatmap shows the density of resulting policies, with color intensity indicating the number of action label pairs yielding each initial policy. The diagonal represents valid policies where p(wa, (c) + p(Wa2lc) zs 1 (minimal illegal actions).
responds to gemma-2-2b-i'ss output distribution over action tokens We fine-tune Llama-3.2- IB-Instruct this dataset using a KL divergence loss that minimizes the difference between the model's output distribution and the target distribution. The action labels used in the dataset (Wa, = =C, D) are identical to the ones used for the baseline and shaping experiments.
Figure (0(c) shows the initial policies of Llama-3.2-12 1.--------t-- after SFT Despite being trained only on examples with action labels Wa, C, Waz D. the fine-tuned model generalizes across action label pairs, producing distributional signatures similar to gemma-2-225 2-2b-it. The initial policies now concentrate near the diagonal (minimal illegal actions) and exhibit state-dependent distribu- tions. The one exception is the stateless condition, which shows a stronger bias toward higher cooperation probabilities.
14
We conduct baseline and shaper experiments as per Section 5.1 with the only difference being the adapter initialization (SFT weights for Llama-3.2-1 1-Instttttt vs. random initialization for gemma- 2-2b-ir). The training prompts maintain the same structure with two model-specific modifications. First, we use the appropriate chat formatting tokens. Second, we provide the game rules and payoff matrix as system prompt instead of prompts, all game relevant information was passed as a user message). Figure TT shows the initial training prompt for Llama-3.2 1.ama--..-1B-Aaee I--I-rucc. The dynamic updates of this prompt throughout training remain unchanged with respect to those used in the main experiments (see Appendix A.11).
message (since gemma-2 2-2b-- does not support system
presents the average reward per step for both players in the baseline (two naive learn- ers) and shaper (one shaper vs. naive learner) conditions. In the baseline, both agents converge to mutual defection with rewards of approximately each, matching the gemma-2-2b-it baseline re-
Table 17
We extract gemma-2 2-2b-it''s probabilities for generating each of the action labels (Wa, C, Way D), renormalize them, and construct sparse target distribution with all other vocabulary tokens receiving proba- bility O.
"SFT was performed using QLoRA with the same adapter configuration (rank-2) used in subsequent base- line and shaper experiments.
26

---

Published as a conference paper at ICLR 2026
Initial prompt for Llama-3.2-IB-nnsttee for the IPD with Way = C, Waz D
</begin_of _text |><] [tcett///p/ssttthhaaeeeedddd///d httpssattteeeeensseeeeeeeeeeeeee Send_header__ce
You playing 2-player game with actions:
Points assigned
follows: c/c: 3/3, C/D: 0/4, D/C: 4/C, D/D: 1/1, Please only use с and p to xeply.<lect.id)>> <lataat bu bed
Choose action for the round. Reply only with
Figure 11: Initial Llama-3.2- 3.2--IB-Instruc prompt for the IPD. The prompt is identical to that of gemmma-2-2bii except for the formatting tags, and the fact that the game description is given as system prompt.
Table 17: Post-training evaluation results for the IPD comparing baseline versus shaper-naive learner pairs using Llama-3.2-1B 3.-s-IB- as the base model. Average rewards per step are reported with 95% confidence intervals across 5 and 10 random seeds for shaping and baseline experiments, re- spectively. Transitions with amll are excluded from the analysis (~ 0.1% and 0.0% of actions respectively).
Baseline
One Shaper
Player
Player 2
Shaper
Opponent
IPD
1.04 + 0.05 0.99 + 0.02
3.97 + 0.01
0.10 - 0.2
sults. With shaper present, the shaper achieves 3.97, while limiting the opponent to 0.10. These outcomes closely replicate those obtained with gemma-2-2b-it, providing preliminary evidence of ShapeLLMMs generalization across model architectures. Training dynamics are shown in Figure 12
Average Reward per step
Baseline
One Shaper
Player
2.5
Player2
2.0
Shaper
2
Opponent
1.0
50
100
150
2.00
50
100
150
200
1.00
1.00
State Visitation
0.75
0.75
CD
0.50
DC
0.50
DC
DD
0.25
0.25
0.00
0.00
50
100
150
200
0
50
100
1.5
200
Epoch
Epoch
Figure 12: Average reward per step (top row) and state visitation (bottom row) during training for the baseline and shaper experiments in the IPD with Llama-3.2-1BB 1---tutnstt base model. In the state visitation figures, the outcome "T" encompasses all transitions which either player chose Gpull- The results reported along with 95% confidence interval over 5 random seeds.
27

---

Published as a conference paper at ICLR 2026
For reproducibility, report the hyperparameters used in these experiments. Naive learner hyper- parameters follow Table 7 except for the value function coefficient, which was set to OvF = 10 reduce instability. This modification applies to both baseline agents and to the opponent in the shap- ing experiments. Shaper hyperparameters follow Table 8 with Ir = 3.41 x 10- eyr = 10-1, and cp 0.1.
A.10 TRAINING DYNAMICS
In this section we present the average reward per step and state visitation throughout training for various experiments discussed the main text. These figures complement the evaluation results by showing how agent behaviors evolved during the learning process. Figure 13 presents the train- ing dynamics for the baseline experiments across the IPD, IMP and ICG (Section 5.1p. Figure [4 shows the baseline training dynamics for the cooperative shaping experiments (Section 6). Finally, Figures [5 and 16 present the training dynamics for shaper experiments against three different op-
ponent types across the IPD, IMP, and ICG, with evaluation results reported in Section 5.2
Average Reward per step
IPD
IMP
ICG
Player Player
0.5
Player
2.5
Player Player 2
Player
2.0
1-5
90
100
150
200
150
200
50
State Visitation
0..8
1HEE
0.75
0.50
0.4
TT
0.50
DD
GO
0.25
0.25
0.00
100
150
200
100
150
200
50
100
150
Epoch
Epoch
Epoch
Figure 13: Average reward per step (top row) and state visitation (bottom row) during training for the baseline experiments across the IPD, IMP and ICG reported Section 5.1 In the state visitation figures, the outcome "I" encompasses all transitions where either player chose O The results are reported along with a 95% confidence interval over random seeds, except the ICG experiment, for which we use 10 seeds.
28

---

Published as a conference paper at ICLR 2026
Average Reward per step
C-IPD
ISH
Player Player 2
Player Player
1.00
1.00
State Visitation
0.75
CC
0.75
SS
SH
0.50
0.50
0.25
0.00
100
200
Epoch
Epoch
Figure 14: Average reward per step (top row) and state visitation (bottom row) during training for
the baseline experiments across the C-IPD and ISH reported in Section 6 In the state visitation figures, the outcome "I" encompasses transitions where either player chose Onall. The results reported along with a 95% confidence interval over 10 random seeds.
29

---

Published as a conference paper at ICLR 2026
IPD
IMP
ICG
1.0
PNL (a1) ~ 0.75
Shaper
Shaper Oppon.cot
Shaper Oppanent
0.0
Opponent
-0.5
1.0
1.0
3.0
P PNL (a1) ~ 0.50
2.5
Shaper Upponent
Shaper Oppon.coi
2.0
Shaper Oppanent
-0.5
1.5
-1.0
1.50
1.0
3.0
PNL (1) - 0.25
0.5
2.0
Shaper
0.0
Shoper
Opponent
Opponeni
-0.5
1.0
Shaper
-1.0
Opponent
50
150
200
100
200
50
150
200
Epochs
Epochs
Epochs
Figure 15: Average reward step during training for the shaping experiments across the IPD, IMP, and ICG with different opponent initializations (Section 5.2 Each row corresponds an opponent with different initial probability of generating action ("Cooperate" in the IPD, "Heads" in the IMP, and "Swerve" the ICG). The results are reported along with a 95% confidence interval over 5 random seeds.
30

---

Published as a conference paper at ICLR 2026
IPD
IMP
ICG
IXII
PNL (a1) 0.75
0.8
T11
0.6
1T
SG
0.2
0.
0.0
1.0
HH
PNL (a1) 0.50
0.8
CC
TH
85
CD
0.6
1T
SG
0.4
0.4
0.2
0.2
0.0
p (a1) ~ 0.25
0.8
0.5
CC
TH
S5
CD)
0.6
TT
0.6
8G
0.4
0.2
0.2
0.0
Epoch
Epoch
Epoch
Figure 16: State visitation during training for the shaping experiments across the IPD, IMP, and ICG with different opponent initializations (Section 5.2). Each row corresponds to an opponent with different initial probability of generating action an ("Cooperate" in the IPD, "Heads" in the IMP, and "Swerve" the ICG). For all games, the outcome "I" encompasses all transitions where either player chose anall. The results are reported along with 95% confidence interval over random seeds.
31

---

Published as a conference paper at ICLR 2026
A.11 TRAINING PROMPTS
We use three types of prompts throughout training: the base prompt, the state-only prompt, and the state occurrence prompt.
The base prompi (Figure 17 includes game description specifying the number of players, available actions, and payoff structure. It also contains an instruction select an action using the designated action labels Ways Waz.
The state-only prompi (Figure 18 includes all information from the base prompt and additionally incorporates the last joint action of both players. Naive learners receive this prompt for all game rounds except the first, where they receive the base prompt. Shapers receive this prompt only the second round of each trial, when one previous action has been played.
Lastly, the state occurrence prompt (Figure 19] includes all information from the state-only prompt and additionally incorporates the state counts for all previous rounds within the current trial. Shaping agents receive this prompt for all interactions beyond the first two rounds. The prompt is dynamically updated as follows: when new actions played, the state count is updated to include the most recent state, and the new joint action displayed the current state field.
Base prompt for the IPD with War = C, W/ = D
<bos><start.offturmueee
You are playing 2-player game with actions: C, D. Points are assigned as follows: C/C: 3/3, C/D: 0/4, D/C: 4/0, D/D: 1/1.
Choose an action for the current round, Reply only with or D. <end.of_turn>
<staatt.oftarrmmodell
Base prompt for the IMP with Wa, H. Way
T
<boe><start of turnruser
You are playing a 2-player game with actions: H, т. Points are assigned as
follows: E/H: 1/-1, H/I: -1/1, T/H: -1/1, T/T: 1/-1.
Choose an action for the current round. Reply only with H or I.
<end of turn>
<start.o of .ttannmmdee
Base prompt for the ICG with tha = S, Was = G
<bos><start.. of turnpuser
You are playing a 2-player game with act i ons: 8, G. Points are assigned as follows: S/S: 2/2, 8/G: 1/3, G/S: 3/1, G/G: -5/-5.
Choose an action for the current round. Reply only with or G. kend.of.turn>
<start.of.turnamodd
Base prompt for the ISH with Va = S, Way
<boa><start wtw.ooo..uuuuooeeeoooooooo turn>user
You are playing a 2-player game with actions: 8, Н. Points are assigned as follows: S/S: 4/4, 8/H: 0/3, H/S: 3/0, H/8: 1/1.
Choose an action for the current round, Reply only with or H.
<end of turn <start of turnsmodel
Figure 17:
Base prompts for the IPD, IMP, ICG and ISH. The structure remains the same across games, with the only differences being the action labels and reward matrices.
32

---

Published as a conference paper at ICLR 2026
Example of state-only prompt for the IPD with Wa, C, Waz D
<boa><start.o..tuznuu
You are playing 2-player game with actions: C, Points are assigned as follows: C/C: 3/3, C/D: 0/4, D/C: 4/C, D/D: 1/1.
<STATR>Tn the previ ous round, you played C and your opponent played C. Choose an action for the current round. Reply only with D. <end.of.turn> <start..o..urmmmooell
Example of state-only prompt for the IMP with Wa. H. Waz T
tbsxxstartoo.tuuuuuue
You aze playing a 2-player game with actions: H, т. Points are assigned as follows: R/H: 1/-1, R/T: -1/1, T/H: -1/1, T/T: 1/-1.
<STATE>Tn the previ ous round, you played # and your opponent played H. Choose action for the current round. Reply only with T. kend_of.tuzy
<start..ff.armmmoods
Example of state-only prompt for the ICG with Wa, Way G
<boa><start of
G. Points are assignes as follows: S/S: 2/2, S/G: 1/3, G/S: 3/1, G/G: -5/-5.
You are playing 2-player game with actions:
<STATE>In the previ ous round, you played and your opponent played S. Choose an action for the current reund. Reply only with or G. <end.of.turn> <starttoo.tarmmmmddll
Example of state-only prompt for the ISH with was S. ta
<bos>><etrt_off fturuuuuerrr
You are playing a 2-player game with actions: S, н. Points are assigned as follows: S/S: 4/4, S/H: 0/3, H/S: 3/0, H/H: 1/1.
<SIATE>In the previous round, you played s and your opponent played 5. Choose an action for the current round, Reply only with or Н. send.of.turn> <start.of.turm
Example state-only prompts for the IPD, IMP, ICG and ISH. The structure remains the same across games, with the only differences being the action labels and reward matrices. The specific examples shown are for rounds in which the previous joint action is (a1,a1).
Figure 18:
33

---

Published as a conference paper at ICLR 2026
Example of state occurrence prompt for the IPD with wai C, Woz D
<bos><start.of wwrrttoo.turruserr.. turn>user
You are playing a 2-player follows: C/C: 3/3, C/D: 0/4, D/C: 4/C, D/D: 1/1.
with actions: D. Points are assigned as
<ADDITIONAL INFORMATION~The occurrence of each state in the current game has been CC:0, CD:0, DC:0, DD:5.
<STATE>In the previous round, you played o and your opponent played c. Choose an action for the <end.ofiturn> catarttoo..trmmmmddll
round. Reply only with
Example of state occurrence prompt for the IMP with Way H, Waz
<bos><start.of_turn
You are playing a 2-player game with actions: II т. Points are assigned as follows: E/П: 1/-1, n/T: -1/1, T/l: -1/1, T/T: 1/-1.
<ADDITIONAL INFORMATION-The occurrence of each state in the current has been нH:C, HI:0, TH:0, II:5.
<SIATE>In the previous round, you played and your opponent played H. Choose an action for the current round. Reply only with or T. <end of.tur <start.o caarttoo..turnmmod
Example of state occurrence prompt for the ICG with Way = S. Wlg G
<bos>>sstart.of -trrnuuer
You are playing a 2-player game with act ons: 8, G. Points are assi gnsi as follows: 8/S: 2/2, 8/G: 1/3, G/S: 3/1, G/S: -5/-5.
<ADDITIONAL INFORMATION~The occurrence of each state in the current game has been 59:0, SG:0, GS:C, GG:5.
<STATI>In the previous round, you played and your epponent played 5. Choose an action for the current round. Reply only with G. <endLof-turn> <start.of.trrnmmode
Example of state occurrence prompt for the ISH with Wa, S, Way H
<boxxstart.of.uurvuss:
You are playing a 2-player game with actions: S, H. Points are assigned as follows: 8/S: 4/4, S/H: 0/3, R/S: 3/0, H/H: 1/1,
<ADDITIONAL INFORMATION:The occurrence each state in the current qame been 88:0, SH:0, H8:0, HH:5,
<STATI>In previous reund, you played your opponent played 5. Choose an send_ofiturns sstartt.f..ormmmoodel
current reund. Reply only with
Example state-oceurrence prompts for the IPD, IMP, ICG and ISH. The structure remains the same across games, with the only differences being the action labels and reward matrices. The specific examples shown for rounds in which all previous joint actions within the game (az, a2), except the last one, which is (11,41).
Figure 19:
34


Reviews:
[{"review_id": "g6lfZuS12b", "reviewer": "Reviewer_ypQM", "summary": "The paper studies opponent shaping (OS) in Large Language Model (LLM) agents. It examines whether an LLM can influence another learning agent’s training dynamics and subsequent behavior solely through repeated interaction. The work introduces ShapeLLM, an adaptation of model-free opponent shaping ideas inspired by M-FOS and SHAPER. Because existing OS algorithms rely on higher-order gradients or architectural components not present in transformer models, ShapeLLM instead encodes both within-episode history and across-episode opponent-learning information as structured natural-language context.\n\nEmpirically, the authors show that LLM agents trained with ShapeLLM:\n- Competitive settings: guide co-players toward exploitable strategies in repeated 2×2 games (Iterated Prisoner’s Dilemma, Matching Pennies, Chicken), yielding higher long-run payoffs.\n- Cooperative settings: induce more cooperative play and higher joint payoffs in Iterated Stag Hunt and a cooperative variant of the Prisoner’s Dilemma.", "questions": "1. Can the authors report ablations on the prompt design? In particular, how does shaping performance vary when altering the amount of intra-episode history versus inter-episode context included?\n2. What computational resources (GPU hours, etc) are required to train the shaper under PPO in the reported settings? And what are the estimation if doing larger scale experiments?\n3. How do the authors expect ShapeLLM to extend to larger or partially observable environments with richer action spaces (e.g., text-based negotiation or resource allocation)? Are there preliminary results or arguments supporting such transfer?\n4. Have alternative base LLMs (e.g., different sizes or pretraining sources) been tested, and if so, how does the choice of model affect shaping behavior? This would help determine whether shaping is model-agnostic.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The work is the **first** formal study of OS in LLM agents. It targets multi-agent LLM interaction concerning how one agent may influence another’s learning dynamics.\n- The proposed ShapeLLM adapts model-free OS ideas to transformer-based agents by encoding within-episode history and cross-episode learning context in structured natural-language prompts. \n- Experiments span multiple repeated 2×2 settings: competitive (Iterated Prisoner’s Dilemma, Matching Pennies, Chicken) and cooperative (Iterated Stag Hunt, cooperative IPD). This shows that the shaping procedure functions under varied incentive structures.", "weaknesses": "- Many works already evaluate LLMs and non-LLM agents in competitive or cooperative games. The paper’s contribution is not fully disentangled from these prior settings.\n- ShapeLLM encodes within-episode and across-episode information as natural-language context. However, the causal link between this prompt structure and the opponent’s parameter updates is not analyzed. As a result, it is unclear to what extent the observed shaping behavior arises from the LLM’s strategic adaptation versus different prompt design.\n- This paper only analyze 2×2 settings, which might limits some claims on competitive setting and collaborative setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HIrXPfzgel", "reviewer": "Reviewer_ZruM", "summary": "The paper investigates a method for fine-tuning small LLMs for opponent-shaping in iterated matrix games using a Shaper-style algorithm where a naive learner updates their parameters inter-trajectory with PPO and the shaping LLM learns after a seeing a full batch of trajectories, called a Trial. Crucially the LLMs only have available the past action tuple and a histogram summary of the past actions in the episode/Trial.", "questions": "1. Are the Naive Learner parameters reset after each Trial? this is not clear to me from the paper.\n2. I'm skeptical of the IPD results in figure 2, the learning does not appear to converge to a Nash equilibrium given that the visitations of DC are by far the most dominant. It seems to me like the naive learner is simply not learning to defect correctly, all past shaping methods on the IPD have the shaping agent converge to tit-for-tat or other reciprocal policies, so it's strange to me that this would converge to always DC\n3. Minor quibble in the introduction, but the distinction between an \"RL agent\" and an \"LLM agent\" does not really exist, RL is a training procedure that can and does produce LLM agents.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "This is the first paper which finetunes LLMs for opponent-shaping in iterated general-sum matrix games. I found the paper decently clear and well-written, the figures and experiments were all well-motivated. The experiments were sufficient to show what the authors wanted to show (especially with what was likely a constrained compute budget). Passing a summary of past actions as context to the LLM is a clever idea to get around the compute constraints of having to pass very long sequences to the models.", "weaknesses": "Unfortunately the biggest weakness is the lack of novelty. \n\nThe claim in the introduction that past opponent-shaping methods cannot be applied to LLMs is wrong. Advantage alignment ( https://arxiv.org/abs/2406.14662 ) changes the PPO advantages in a way that would also be easy to do for LLMs. And the following paper from earlier this year also implements a meta-episode context with transformers which implements opponent-shaping: https://arxiv.org/pdf/2410.18636 . Past papers don't implement the summarization mechanism for giving context to the model, but this is a trick for saving compute that only works in toy contexts like iterated matrix games, for more complex games one would need to pass the entire past N trajectories as context, which is exactly what Meulemans et. al. do.\n\nThe novelty of the paper lies entirely in applying a more limited version (because of the summarization) of the Meulemans et. al. idea to finetuning LLMs on a simpler game than they have.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NF3nCaOfWT", "reviewer": "Reviewer_X6KR", "summary": "This work proposed an opponent shaping method (ShaperLLM) and applied it to multi-agent game playing tasks to invesitgate how opponent shaping impacts agents' behaviors. ShaperLLM's algorithm derives from SHAPER by Khan et al., and the authors simplified it and directly optiomized LLMs with Reinforcement Learning. The experiments highlight that LLM agent itself can perform opponent shaping and opponent shaping can promote agents' cooperation.", "questions": "1. I would like to see clear description of the **formula definitions** & **optimization objectives** of opponent reward and shaper reward in Section 3 (line 212-215).\n2. Somes terms are interchangeably used and confusing. For example, I think $t$ in line 162 means each step, but in later context (line 174-175) means one round. The meaning of $n_{games}$ is not straightforward, does it mean the total number of games? Maybe use one letter like $N$ to replace $n_{games}$ is better. Also in line 174-175 you mentioned *trails* and in line 214-215 you addresses *trail return*, but in experiment setup (Section 4.2) the trail setting is not described; you described *step* instead. What's the relation between *trail* and *step*, could you unify/distinct the terms for the same/different concepts?\n3. In line 199, $POMDP(...)$, what's the meaning of $\\Omega$ and $\\gamma$?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Proposed a method that directly incorporating SHAPER algorithm into the training process of LLMs without external policy network like RNNs.\n2. Conducted experiments on various game playing tasks and presents insights on the effectiveness of opponent shaping on multi-agent scenario.", "weaknesses": "1. **Presentation of ShaperLLM's algorithm lacks clarity**: In section 3, the description of two RL rewards (opponent reward and shaper reward) of ShaperLLM is unclear. The two rewards are heavily analyzed in the experiments but their formulas are lacking. Also, some terms for the setting description are a little confusing, like *steps*, *trails*, *rounds*, as they are often interchangeably used in different contexts.\n2. **Experiments lack generalizability and scalability**. This work only did experiments on gemma-2-2b-it so their findings may lack generalizability (the authors also ack this in the paper).\n3. **Weak baseline**. The work compares ShaperLLM only with *naive agent* that sees only one step of previous interaction history and doesn't compare with other RL methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DTemxvupaj", "reviewer": "Reviewer_Dj8k", "summary": "The paper introduces ShapeLLM, a model-agnostic opponent shaping (OS) algorithm that is suitable for LLM architectures. ShapeLLM leverages natural language capabilities by encoding history and context in the repeated play in natural language prompts. It then utilizes a partially observable stochastic game formulation for the policy updation process, making it potentially useful even beyond the context of repeated normal-form games. The effectiveness of the proposed algorithm is demonstrated in a series of 2*2 repeated games, involving both competitive and cooperative interactions. The results demonstrate that in these particular games, ShapeLLM improves the performance of the shaper agent in the competitive games, and steers the interactions towards collaboration in cooperative games.", "questions": "I don’t have a specific question here, but I would be interested in the authors’ reflections on how far they believe their approach could generalize to more complex and realistic scenarios. It would also be valuable to hear their thoughts on potential limitations or pitfalls in such extensions, and how these challenges might be addressed.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is very well-written, and is very easy to follow and fun to read. The paper covers the related literature, making it easy to understand its contribution to the existing research landscape. It also provides preliminaries for readers who are not entirely familiar with some aspects of the work, making the writing very suitable for the ML community, which is fast-growing and multi-disciplinary by nature.\n\n2. The paper proposes a novel approach that addresses crucial gaps in the existing OS literature. In particular, the proposed approach can be naturally applied to LLM-based agents in various contexts, and as the paper itself suggests, even far beyond those introduced in the experimental setup.\n\n3. The paper reports experimental results using classical game-theoretic setups that capture core elements in strategic behavior that are crucial for OS. This simplicity enables the identification of core behaviors that emerge using the proposed algorithm, making it easier to understand not only *whether* OS occurs, but also *how*.", "weaknesses": "My major concern with the paper (which was also discussed by the authors themselves) is the oversimplicity of the experimental setups, which only include simple two-player two-action games, with no natural language communication. While this simplicity is obviously an advantage (see the *strengths* section), I believe that providing some results in some more complex and realistic environments could have significantly strengthened the contribution of the paper, especially for ML practitioners who might want to apply it to real-world scenarios such as trading, negotiation, persuasion, content generation, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)