Towards Sustainable Investment Policies Informed by Opponent Shaping
Authors: Juan Agustin Duque
Razvan Ciuca
Ayoub Echchahed
Hugo Larochelle
Aaron Courville
Venue: ICLR 2026 Poster (2026)
Keywords: Climate Change
Social Dilemmas
Multi Agent Reinforcement Learning
Opponent Shaping
Policy Making
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We determine the conditions in which InvestESG, a MARL simulator of investments under climate risk, is a social dilemma. Then we apply opponent shaping to it and provide insights on how the resulting policies can inform policy making.

Abstract: Addressing climate change requires global coordination, yet rational economic actors often prioritize immediate gains over collective welfare, resulting in social dilemmas. InvestESG is a recently proposed multi-agent simulation that captures the dynamic interplay between investors and companies under climate risk. We provide a formal characterization of the conditions under which InvestESG exhibits an intertemporal social dilemma, deriving theoretical thresholds at which individual incentives diverge from collective welfare. Building on this, we apply Advantage Alignment, a scalable opponent shaping algorithm shown to be effective in general-sum games, to influence agent learning in InvestESG. We offer theoretical insights into why Advantage Alignment systematically favors socially beneficial equilibria by biasing learning dynamics toward cooperative outcomes. Our results demonstrate that strategically shaping the learning processes of economic agents can result in better outcomes that could inform policy mechanisms to better align market incentives with long-term sustainability goals.

Published as a conference paper at ICLR 2026
TOWARDS SUSTAINABLE INVESTMENT POLICIES IN- FORMED BY OPPONENT SHAPING
Juan Agustin Duque", Razvan Ciuca", Ayoub Echchahed', Hugo Larochelle', Aaron Courville' University of Montreal and MILA firstname lll.l name umonnreallcc
Eqqua contribution
'quual supervision
*Core contributor
ABSTRACT
Addressing climate change requires global coordination, yet rational economic actors often prioritize immediate gains over collective welfare, resulting in social dilemmas. InvestESG is recently proposed multi-agent simulation that captures the dynamic interplay between investors and companies under climate risk. We pro- vide formal characterization of the conditions under which InvestESG exhibits intertemporal social dilemma, deriving theoretical thresholds at which individual incentives diverge from collective welfare. Building on this, we apply Advan- tage Alignment, a scalable opponent shaping algorithm shown to be effective in general- sum games, influence agent learning in InvestESG. We offer theoretical insights into why Advantage Alignment systematically favors socially beneficial equilibria by biasing learning dynamics toward cooperative outcomes. Our results demonstrate that strategically shaping the learning processes of economic agents can result in better outcomes that could inform policy mechanisms to better align market incentives with long-term sustainability goals.
INTRODUCTION
Climate change is one of the defining challenges of our time, which demands immediate, coordinated global action to mitigate catastrophic environmental and economic consequences (Pörtner et al., 2022). Despite international agreements such as the Paris Agreement, nations continually grapple with aligning individual economic incentives with collective environmental goals. This misalignment can be effectively modeled as a social dilemma: a scenario in which players, acting independently and rationally according their own interest, produce suboptimal collective outcome (Rapoport and Chammah, 1965). In climate negotiations, nations often prefer short-term economic benefits over the collective welfare of aggressive mitigation strategies, leading to Pareto-suboptimal equilibria.
Previous advances in reinforcement learning (RL) have demonstrated remarkable capabilities in optimizing agent behaviors in various complex tasks (Mnih et al., 2013; Schulman al., 2017). However, when confronted with general-sum games such as climate change negotiations, traditional RL methods consistently converge to selfish strategies that maximize immediate individual rewards but result in globally suboptimal outcomes (Sandholm and Crites, 1996; Foerster et al., 2018). These greedy equilibria do not realize the potential gains achievable through cooperation, underscoring the critical need for methods that inherently incentivize collaborative behavior among rational players.
To address this fundamental limitation, opponent shaping (Foerster et al., 2018) has emerged as promising approach, allowing players to strategically influence the learning dynamics of other participants. Advantage Alignment, a recent advancement within opponent shaping algorithms (Duque al., 2025), operates by aligning the advantages of interacting players. Unlike earlier opponent shaping methods that rely on computationally expensive imagined parameter updates (Foerster et al., 2018; Willi et al., 2022), Advantage Alignment modifies policy gradients directly, promoting actions that benefit both individual and collective outcomes efficiently.
Conventional MARL methods (IPPO (Yu et al., 2022)) optimize myopic returns, while recent opponent-shapine approaches (LOLA (Foerster et al., 2018). M-FOS (Lu et al., 2022). BRS Agha- johari al. (2024a)) scale poorly beyond simple games or limited to discrete action spaces

---

Published as a conference paper at ICLR 2026
(LOQA (Aghajohari et al., 2024b)). In this paper, we apply Advantage Alignment in the context of sustainable investment policies in the InvestESG environment (Hou et al., 2025), a realistic simulation of corporate and investor interactions under climate risk. Specifically, contributions
follows.
We formally prove for a simplified InvestESG that the parameter a (the climate responsive- ness mitigation) critical for making the game a social dilemma in the stochastic setting (see Appendix D), and empirically show that the full game becomes a dilemma with a = 70.
We adapt Advantage Aligmment to handle the complex dynamics characteristic of the original InvestESG simulation, without any simplifying assumptions, demonstrating its scalability and effectiveness compared to IPPO and MAPPO baselines (Yu et al., 2022).
We provide a theoretical argument for why Advantage Alignment finds better equilibria than other baselines (including PPO with incentives and PPO summing the rewards).
We provide insights into the policies discovered by Advantage Alignment, which contribute mitigation more effectively than other algorithms.
2 BACKGROUND
2.1 MARKOV GAMES
We consider the formalism of n-player, fully observable, general-sum Markov Games (Shapley, 1953) which are represented by a tuple: .M (N, U,N,S,,, P, R, 7). Here, the state space is denoted S. The joint action space for all players in the game is written as AA := AA' x x A"" The transition function, P: S A + (SS), maps from every state and joint action to distribution over states. The reward structure is given by R = {r'..... r where each r² S x A + R specifies the reward received by player The discount factor, 7 E [O, 1], determines the relative importance of future rewards.
2.2 REINFORCEMENT LEARNING
Consider n players interacting in an environment with policies n², with E [l...., n], each parame- terized by 0'. Here, the result for each player is determined not only by their own actions but also by those of other players (von Neumann and Morgenstern, 1944). Following the convention Agarwal al. (2021), we can write the probability of trajectory 7 under the policies of all players follows: Pr . (-'((7) (7) = μuss)) II==1 (a(a)ss)) P(s ao) Where μ denotes the initial distribution over states, and P(.|s, a) the transition distribution over states, which depends on the current state, 8, and the joint action profile of all players, a. In reinforcement learning the goal is to maximize the expected sum of future discounted reward, i.. the value function:
V'(n) = E 7- --Pp"-a- [R' = E
Σ (tt at)
We also define the action-value function, also known as the Q-value, to define the advantage function:
(s, a) = E -- R (7) so = s, ag aa] A'(s, a) = Q' V'(s).
(2)
To optimize the value function, policy gradient methods perform gradient ascent the policy parameters following a REINFORCE estimator (Williams, 1992) of the form:
V (1) E T.Pr
Σ'' A' a Vo: log n' (a()
The most popular variant policy gradient algorithms, Proximal Policy Optimization (PPO) (Schul- man et al., 2017), makes conservative updates to the policy using clipping heuristic:
(aaa 81; 0*) n (a s1; 0))
n (a(aa be; 0¹) (aa se; 07)*
min
A' ax), clip
e, 1.1+c) +c) A A an)
(4)
T
Where 0j denotes the policy parameters of player after gradient updates. Intuitively, PPO attempts to keep the updated policy close to the original policy in terms of total variation distance.
2

---

Published as a conference paper at ICLR 2026
2.3 SOCIAL DILEMMAS
Social dilemmas, as introduced by Rapoport and Chammah (1965), describe multi- multi-player player scenarios in which individual rational play leads to outcome with less utility than that which would have occurred if all players had cooperated. Formally, consider a normal-form game with n players. where each player selects an action a' from their action space A', and receives utility (a ..... a²). Let ac = (ac....,"") a²) denote joint cooperative strategy profile that maximizes the total social welfare.
Definition (Nash Equilibrium (Nash, 1950)). Consider an n-player game where each player i E (1,.... , n has action space A' and payoff function r' AA' x... A R. A strategy profile aN = (a):..... ,.x)) is Nash equilibrium for all players i:
1 (d'x, ax') 2 ('(d'aan) Va' E EAA'
(5)
a Here, aN denotes the strategies chosen by all players other than player i.
In this work we treat climate negotiations as a social dilemma. Informally, a repeated interaction social dilemma when there exists cooperative pattern of behaviour that maximizes collective welfare, yet each individual actor has a short term incentive to deviate and free ride on the efforts of others. In such settings, myopic best responses tend to erode cooperation over time, agents would be better off if they could sustain it. A formal definition social dilemmas Matrix games given in Appendix A. For analyzing social dilemmas Markov games, we define the price of anarchy (Nisan et al., 2007).
though all
Definition (Price of Anarchy in Markov Games), Let M be an n-player Markov game. Denote by II the set of joint policies and by N C II the subset that are Nash (Definition 1). For a joint policy a E lei ((T; p)) = Σ (H) the social welfare function. The price of anarchy
max W(T:p)
Telf Pa
(6)
min W(T:))) TEN
Markov game exhibits a social dilemma whenever Pa > i.e. the worst equilibrium achieves strictly lower total return than some feasible, W-maximizing, joint policy. Intuitively, the price of anarchy the ratio between the welfare of coordinated and individual rational play.
2.4 OPPONENT SHAPING
Opponent shaping, introduced by Foerster et al. (2018), changes the paradigm of reinforcement learning: instead of assuming stationarity of the environment, opponent shaping algorithms assume that other players are learning agents. By making assumptions about the learning algorithms (Foerster et al., 2018) or modeling the problem as a meta-game (Lu et al., 2022), opponent shaping algorithms attempt to influence the learning dynamics of other players. Duque al. (2025), proved that controlling other players via the Q-values is equivalent to doing policy gradient with the following modified advantage:
A ar) A'(s +
Σ E au) AeCt
A Al (8)
This Advantage Alignment formula can be plugged into the PPO formulation for a scalable Opponent Shaping algorithm (Proximal Advantage Alignment) that works in complicated multi-agent settings with high dimensional state representations, continuous action spaces, partial observability and multiple (more than 2) agents.
INVESTESG
InvestESG (Hou al., 2025) is multi-agent reinforcement learning (MARL) environment designed to study the long-term impact of corporate climate investments and ESG (Environmental, Social, and Governance) disclosure mandates in an intertemporal economy under climate risk. The environment
3

---

Published as a conference paper at ICLR 2026
simulates interactions between company agents (who allocate capital across mitigation, resilience, and greenwashing strategies) and investor agents (who reallocate capital across companies based on profitability and ESG scores). Climate risk evolves over 100-year horizon, shaped by cumulative mitigation efforts, and materializes through climate events that inflict economic losses on companies. By default, InvestESG uses companies, 3 investors and doesn't 't allow for resilience or greenwashing strategies, although these parameters can be modified. The main limitations of this simulator, as result of this parameterization, are clearly stated in the InvestESG paper (Hou al., 2025).
3.1 ECONOMIC AND ENVIRONMENTAL DYNAMICS
For notational convenience, we use the index i E [1...., M] to refer to companies and index j E [[.....N] to refer to investors. InvestESG, each investor's state is represented by their cash level C. and investment portfolio S = (HH, Hi where Hi, denotes the investment that investor j holds in company at timestep t. At each timestep, investors pick a binary portfolio vector af = lo.,...., d(,arl representing which companies to invest in and companies choose to mitigate a percentage of its capital, n/ [0, ūj. The interim capital of each company at time t + 1 is given by:
N
Ki S+ Linterim
Ki
(8)
a
J=1
cash returned to investors
new equity investments
Where Kj denotes the capital of company i and K2 the capital of investor j at time f. The reward each timestep for company is given by the profit margin, rf K K interim Climate dynamics, namely the risk that particular type of harmful climate event occurs Pe (where e can be either an extreme heat, h, heavy precipitation, p, or drought, d, event) grows linearly w.r.t. initial event probability Pj following:
Mel
P
+ Po; == de,
1+ XeU,
Where He, Åe, are calibration constants set ensure that an annual mitigation investment of 2.3 trillion is required to achieve IPCC's 1.5ºC scenario (Hou et al., 2025). We add an additional calibration constant, that scales the À parameters in order to make InvestESG a social dilemma (for details see section 4). The total risk P. mitigation U, is:
-1- P1) P,") P and the cumulative
M
U. = U, +
Σ
u Kit 1,+1,imerrm
(10)
2=1
Finally, the capital company i is updated as follows:
K2t1 (1 pt) (41,
(1-1 7)(1 X,I,) 1.
(11)
Where, of is the profit margin of company i. Each type of harmful climate event (extreme heat, heavy precipitation or drought) is modeled independently by Bemoulli(P?)) where X, E f0, 1, 2, 2. is the total number of climate events realized at time L, above denotes the loss coefficient of company and y (which differs from the discount factor) is the market performance baseline.
4 A FORMAL ANALYSIS OF INVESTESG AS SOCIAL DILEMMA
This section contains our main thcoretical contribution: we show that, in InvestESG, the climate responsiveness parameter a is the critical quantity that determines whether the game behaves as social dilemma, and we identify the corresponding threshold regime. Originally introduced social dilemma, the InvestESG environment is here studied the stochastic setting with unseeded environmental dynamics, where observe that the empirical price of anarchy (Definition 2) is close to (Figure 5). The observed equilibrium does not change even when adding extremely strong ESG incentives such that investors effectively only about ESG (the ESG=10 case in the figure). In social dilemma, we would expect to see that adding prosocial incentives changes the final behavior of agents, yet this is not what see empirically.
4

---

Published as a conference paper at ICLR 2026
Mitigation
500
400
0.5
3000
300
2003
0.3
0.2
1000
100
0.0
Status ESG
Status ESS
ESG
Mitigation
Climate
0.
1030
0.3
750
0.2
5030
230
0.0
Status
ESG
ESG
ESG
0.5
Figure Comparison of final environment metrics different a (introduced equation 9) values, for 10 seeds of PPO agents: With the default scaling (a = 1), the differences in market total wealth and final mitigation amount negligible between agents with different ESG incentives. Storus Quo refers PPO agents trained without incentives. Increasing the scaling factor (a 70) results higher market total wealth for ESG-conscious investors, higher price anarchy. The whiskers indicate I-standard deviation confidence interval. It should be noted that the ESG score is a multiplicative parameter, ESG = 10 represents an immense prosocial reward: the investors in that scenario effectively do not care at all about their own profits. These experiments were with the full complexity allowed by InvestESG, without simplifications.
In this section, we derive conditions under which simplified InvestESG explicitly meets the definition of social dilemma, clarifying theoretically when this environment manifests the characteristic tensions between individual rationality and collective welfare. We then validate our theoretical results empirically in the full version InvestESG without simplifications (See Figure 5). We find that the effectiveness of mitigation actions X, namely how much each dollar invested in mitigation affects the climate event probability, is the critical parameter which makes InvestESG a social dilemma. We scale the mitigation effectiveness by multiplying the original parameter À by scaling factor a (equation 9). Identifying this regime is essential, because it mirrors the real-world dimate-mitigatin problem in which near-term incentives myopically outweigh the long-term benefits of coordinated action for self-interested agents (Barrett, 1994). We look at three metrics: (I) Market Total Wealth is the combined value of all companies' capital and investors' cash the end of the simulation: (2) Final Mitigation Investment is the total amount all companies have cumulatively spent on mitigation by that point; and (3) Final Climate Risk is the model-estimated probability that at least one damaging climate event occurs at the final timestep.
4.1 SET-UP AND NOTATION
Consider the amount of mitigation that company invests time t, denoted u., Intuitively, company cares about the marginal effect, benefit, that this mitigation amount will have at the next time-step on their capital, K Since K a random variable that depends on the realized number of climate events X,, we take the marginal effect to be the partial derivative of the mitigation amount w.r.t. the expected value of the capital at the next time-step. See Appendix B for details.
Definition 3 (Private Marginal Gradient). Let ut be the mitigation amount of company time t, then the private marginal gradient is
Kit1
E
K
+
(2 (1, wt)(1 + +
Σ
deHtet
(12)
du
(1 XaU,)
namely, the marginal effect in the capital of company by investing in mitigation time-step t.
5

---

Published as a conference paper at ICLR 2026
300
Final Mitigation
250
200
150
50
10 10
10
10-2
102
10a
Figure 2: Final mitigation amount of seeds of PPO agents trained in single company and single investor environment different values of a. Here the public and private gradients are same and clearly threshold a =u 30 at which the final mitigation amount significantly increases. This threshold empirically marks the change of sign of the gradient. The whiskers indicate I-standard deviation confidence interval.
This gradient indicates whether company has a selfish, myopic incentive invest in mitigation: if the gradient is positive, then the capital at the next time-step increases for each dollar invested mitigating. Similarly, we define the social marginal gradient.
Definition 4 (Social Marginal Gradient). Let ui be the mitigation amount of company at time t, then the social marginal gradient is
d E du
--
B
ΣE [1-
(K XLL uf (1
Σ
Aeplet
(13)
(1+
namely, the marginal increase in the capital of all companies by company investing in mitigation at the next time-step.
Our argument builds from the intuition that a social dilemma arises if there is a fundamental tension between these two gradients: the social marginal gradient pushes companies mitigate, whereas the private gradient does not. The final proof, however, is more involved, as we analyze the effect of all previous mitigation actions the final private and social capitals.
4.2
WHEN IS INVESTESG A SOCIAL DILEMMA?
The full stochastic dynamics of InvestESG are too complicated for exact analytic analysis so we simplify the environment. To isolate the social-dilemin component dimate-mitigation decisions, work under the following assumptions for the remainder of this section.
Assumption 1. (Static Investor Portfolios). Investor holdings are inna-invaiaat. Formally, Hi, Hi Vt 2 0, E [M], j E [N]. Thus investors never rebalance their capital allocations.
Assumption 2. (Single-Company Allocation). Each investor supplies capital to exactly one firm.
Assumption 3. (No Historical Mitigation). Prior to the period of interest, firms have undertaken mitigation expenditure: u', = 0 V < t, i [M].
We are interested in computing the incentives on companies to start mitigating climate change in world where one has ever mitigated before and where the investor dynamics are simplified. These assumptions are not used in the experimental section. In this simplified world, we show that the existence of social dilemma critically depends on the parameter X of the simulation, which the responsiveness of climate change to mitigation investment, We give here an outline of the argument, the full derivation Appendix
1. There exists Noow small enough such that in both the individual and social case, none of the derivatives dE[ //u are positive.
2. In the individual case, there exists À such that at least some of the derivatives are positive, i.e. that it is possible to mitigate for self-interested reasons.

---

Published as a conference paper at ICLR 2026
Market Total Wealth
Final Mitigation Investment
Rinal Climate Risk
0.65
17500
15000
0.00
1250
12500
1000
10000
7300
5000
0.45
10
20
30
60
50
60
70
10
20
30
40
60
70
10
20
30
60
60
70
Environment Steps (Million)
Environment Steps (Million)
Environmennt Steps (Million)
AAA
ESS ESG+1
ESG-10
Statos
Figure 3: Training curves of PPO agents with different ESG values and Advantage Alignment (AdAlign) with 70 for different metries: Advantage Alignment agents achieve highest market total wealth by being more strategic about their mitigation investments compared to PPO agents. With significant lower final mitigation investment, AdAlign agents are able to achieve the same final climate risk and increase their capital returns. The shaded areas indicate I I-standard deviation confidence interval. We note that 0.48 is the hest achievable climate risk in the environment, as corresponds to II. (1 )b the floor of the probabilities of each event.
3. By intermediate-vann theorem there is Aeriseal where at least one of dE[ 11/du/uu
4. For the social case, the derivatives are strictly greater than those in the individual case i.e. company i's mitigation efforts on all other companies is positive.
5. Therefore, there exists À such that self-interested agents will not mitigate despite mitigation being beneficial for social welfare, thus we have a social dilemma.
This suffices to establish the existence of zones in parameter space, defined by the two parameters Alow and Acritiat- For À < Auow, there no dilemma because there is agreement between the signs of the individual and social gradients: mitigation is always net negative. If Atow S X S A the gradient signs of the individual and social returns disagree, and we have a social dilemma. If A Aeritical then there again no strong dilemma and self-interested agents start mitigating myopically. For proofs and formal theorems for each of the steps in the outline see Appendix B. We empirically validate that this threshold exists by conducting an experiment in which we train three seeds PPO agents, keeping the number of companies and investors fixed to one, but sweeping over the scaling parameter a (Figure 2). We find the threshold to be = 70, after running sweeping experiments with the default parameters (see Appendix E) When a increases beyond this threshold, PPO agents with different ESG incentives start show different behaviors (Figure 5). We refer to this specific parameterization of InvestESG, with 70, InveeesEESG.
5 APPLYING OPPONENT SHAPING TO INVESTESG
Having identified regime in which InvestESG behaves social dilemma, next model firms and investors as adaptive learners whose policies co-evolve in response to each other, reflecting that real market participants adjust their behavior over time rather than committing equilibrium strategy. We apply Advantage Alignment (Duque et al., 2025) without ESG incentives a- a-InvestESG (found empirically in the previous section), in centralized training decentralized execution manner (CDTE). Note that a-InvestESSG does not have the simplifying assumptions of the theoretical analysis and has the full complexity of the original simulator. Crucially, our Advantage Alignment implementation differs from the PPO implementation in two aspects. First, we substitute the advantage in the policy gradient with the Advantage Alignment expression (equation 7). Second, following the original paper we use self-play: one set of parameters for the company and another set of parameters for investor players. We found that self-play makes training more stable for Advantage Alignment (AdAlign) agents but hurts PPO agents, thus we compare the best version of each in Figure (AdAlign with self-play, PPO without). Advantage Alignment agents outperform all PPO agents, even those with ESG incentives, while simultancously having the lowest amount of final mitigation investment. For details about the policy parameterization and hyper- parameters used for these experiments refer the Appendix D.
fixed

---

Published as a conference paper at ICLR 2026
3500
Investment Gini Coefficiert clent
1.0
Total Wealth
3000
2500
0.8
2000
1500
0.4
1000
0.2
500
0.0
Number of Compankeslivvestrrs
10
20
30
40
50
60
70
Environment
(Million)
MAPPO
IPPO AdAlign
AdAlign
ESG
BSGm1
ESGm 10
Status quo
(a)
(b)
Figure 4: (a) Final market total wealth 10 seeds of Advantage Aligmment without ESG incentives, and PPO agents trained using summed rewards in InvestESG (a = 70). On the x-axis we increase the number of companies and investors (1 company and I investor, 2 companies and 2 investors, etc.) while keeping the initial capital the same. Sum rewards is unable to find the action profile that maximizes social welfare the number players grows beyond a threshold (> 2), whereas Advantage Alignment consistently finds the same solution once the number of agents large enough (> 1) going up to 10 agents. (b) Gini coefficient measuring inequality among company investments for different algorithms. Lower Gini indicates lower inequality and better distribution of resources. The shaded areas indicate 1-standard deviation confidence interval.
5.1 COMPARISON WITH OTHER BASELINES
Intuitively if we care about social welfare, then we should try to maximize explicitly. However, this does not yield good results in practice. We experiments with summed rewards with the default InvestESG settings (5 companies, 3 investors), however the IPPO and MAPPO (Yu et al., 2022) agents always converged to sub-optimal equilibria. Our rationale for the baseline selection is discussed in the appendix D.3. Figure 4 (a) shows that naively summing the rewards scales poorly in the multi-agent setting. We run 10 seeds of IPPO and MAPPO agents that sum their rewards, scaling the number of agents in the environment, and plot the final market total wealth with initial fixed capital. Beyond companies and investors (4 agents total), PPO players summing the rewards are unable to find policies that maximize the social welfare. We hypothesize that the credit assignment problem becomes more difficult as the number of agents increases, giving noisy temporal difference signals to centralized critic methods like MAPPO. In contrast, Advantage Alignment maintains high social welfare across all experiments. We analyze the reasons why this happens in section 6.
5.2 INTERPRETING ADVANTAGE ALIGNMENT POLICIES
To assess how opponent shaping alters investment behaviors in the social-dilemma setting, simulations of Advantage Alignment (AA) and the standard PPO baseline with ESG score = 0, which means that there no external incentive mitigate. The policies obtained summarized in Fig. 7. Three patterns emerge.
(i) AA learns to mitigate just enough and only at the critical moments when climate risk increases, whereas PPO invests excessively. (ii) The capital allocations show that AA maintains an approximately uniform investment distribution across companies, while PPO concentrates wealth, replicating the disparities reported by Hou et (2025) (Figure 4(b)). (iii) AA agents learn to coordinate themselves to share mitigation costs, in contrast to PPO agents whose independent updates yield uncoordinated, non-cooperative mitigation investments.
ON THE EFFECTIVENESS OF ADVANTAGE ALIGNMENT
Advantage Alignment is particularly effective finding social welfare maximizing strategies in social dilemmas at least in part because it has strong initial bias towards cooperative strategies. When advantages are estimated with Generalized Advantage Estimation (GAE; Schulman et al., 2018), Advantage Alignment nudges agents toward cooperation. For agent i the transformed advantage is
Ar A
8 343
1 Ag.
(14)
k<t
8

---

Published as a conference paper at ICLR 2026
with Ai A' ar).
Define the on-policy mean b' E
E x 2 AAil-
Decomposing the
expectation gives
A5t
+
+
Σ j7
Al.
A b¹) Ag.
(15)
171 K<t
cooperative bias
zero-mean
The first term favors joint gains; if Bybb 1, the cooperative term exactly that encountered in summed-rewart learning. With perfect critics, b' = O, this bias vanishes. However, during actor-critic training, the critic lags the improving policy, so the one-step TD error
ó, re (V++1)) VV((s).
(16)
is positively biased (see Kearns and Singh (2000)) for a formal argument based on contraction mappings): [[ > 0 = [[ > 0 = b' > 0. Hence early training amplifies cooperative updates. As the critic catches up and TD errors become unbiased, 1/ + 0 and the cooperative bias
fades. This initial bias prevents the algorithm from falling into defective equilibria early in training.
7 RELATED WORK
Opponent shaping treats co-learners as adaptive parts of the environment and deliberately steers their updates. LOLA (Foerster al., 2018) introduced the idea by differentiating through single naive policy-gradient step of other players. Follow-up methods refined the update rule (Letcher et al., 2021; Zhao al., 2022; Willi et al., 2022) or approximated best-response dynamics (Aghajohari al., 2024a; Lu et al., 2022). LOQA replaced higher-order differentiation with REINFORCE- based control of opponents' Q-values, enabling scalability to larger games (Aghajohari et al., 2024b). Previous work on Advantage Alignment showed that many of these algorithms implicitly multiply agents' advantages and offered simpler, computationally cheaper formulation that preserves Nash equilibria general- sum games (Duque et al., 2025). The present paper leverages that insight to shape unvestor-company interactions in a realistic Himate-finance simulator. A growing line research embeds MARL agents inside climate models to study emergent behavior, test policy instruments, optimize control strategies. RICE-N extends Nordhaus' regional integrated assessment model into a multi-agent game in which learning agents negotiate emission targets and investment pathways (Zhang al., 2022). The Al for Global Climate Cooperation competition has since used this platform to study how AI negotiators can sustain long-run cooperation under enforcement uncertainty. Our work differs in scope: rather than modeling sivergoovrnmmett bargaining, we focus on the micro-level interaction between profit-secking firms and ESG-motivated investors captured by InvestESG (Hou et al., 2025). Inspired by the A/ Econonis! (Zheng et al., 2021), Wang et al. simulate a cap-and-trade market with enterprise and government agents; the government agent learns dynamic permit-allocation rules that balance output, equity, and emissions (Wang et al., 2024). Our study is complementary: we assume no central planner and instead demonstrate that shaping frrm-investor learning alone can steer the economy toward socially desirable equilibria.
8 CONCLUSION
In this paper, we presented a rigorous theoretical and empirical examination of economic simulations under climate risk, using game theory. By formally characterizing the InvestESG environment as intertemporal social dilemma, we established the precise conditions under which individual incentives diverge from collective sustainability goals. This allowed social dilemma conditions observed in the real world. Leveraging the scalable power of Advantage Alignment, we demonstrated that strategically shaping agent learning dynamics can effectively mitigate selfish incentives, systematically driving the equilibrium toward socially beneficial outcomes, even in the absence of government mandates. Our empirical results underscore the practical efficacy of opponent-shaping algorithms in complex, high-dimensional economic scenarios, addressing critical gap where previous multi-agent RL methods have failed. By elucidating why Advantage Alignment naturally improves the credit assignment problem and finds more cooperative equilibria, we provided key insights that pave the way for robust, policy-relevan solutions capable of maximizing the social welfare more effectively. Beyond theoretical advancements, this work signifies tangible step toward harnessing artificial intelligence to tackle real-world climate policy challenges and inform government decision-making.
fine-tune InvestESG to capture the

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We provide all details needed to replicate our results. The full set of training hyperparameters, model architectures, optimizer choices, rollout lengths, discounting/GAE settings, clipping. and gra- dient/entropy coefficients for all methods (IPPO and Advantage Alignment) are listed Appendix D, Tables I and 2. We also report the number of parallel environments, episode length, total environment steps, and (where applicable) the use of self-play, as well as the number of random seeds for each experiment in the figure captions. Upon de-anonymization, we will publicly release the complete codebase, including experiment configs, training scripts, and plotting utilities to reproduce all figures and tables from scratch.
AKNOWLEDGMENTS
Juan Agustin Duque supported by the St-Pierre-Larrchhe Scholarship the University of Montreal and by Aaron Courville's CIFAR AI Chair in Representations that Generalize Systematically.
REFERENCES
A. Agarwal, N. Jian, S. Kakade, and W. Sun. Reinforcement Learning: Theory and Algorithms. Preprint, 2021.
M. Aghajohari, T. Cooijmans, J. A. Duque, S. Akatsuka, and A. Courville. Best response shaping, 2024a.
M. Aghajohari, J. A. Duque, T. Cooijmans, and A. Courville. Loqa: Learning with opponent q-learning awareness, 2024b. URL https: /arxivv./riivva//aa/eaaeer org/ /rgg/bs/////aaaoo 220/2/455.0020 01035.
S. Barrett. Self-enforcing international environmental agreements. Oxford Economic Papers, 46 (8upplement::878-55 894, 1994.
V. Capraro and J. Y. Halpern. Translucent players: Explaining cooperative behavior in social dilemmas. In R. Ramanujam, editor, Proceedings Fifteenth Conference on Theoretical Aspects of Rationality and Knowledge, TARK 2015, Carnegie Mellon University, Pitssbrgg,, USA, June 4-6, 2015, volume 215 of EPTCS, pages 114-126, 2015. doi: 10.4204/EPTCS.21559 URL https: doi /ttp:::///ooooo//1./.. 4204/EPTCS 215.9.
J. A. Duque, M. Aghajohari, T. Cooijmans, R. Ciuca, T. Zhang, G. Gidel, and A. Courville. Advantage alignment algorithms, 2025. https// //ap:i//xiiviiv/ www.oorgaaaae /aag/aab/// /2w./246... 14662.
J. N. Foerster, R. Y. Chen, M. Al-Shedivat, S. Whiteson, P. Abbeel, and I. Mordatch. Learning with opponent-learning awareness, 2018.
X. Hou, J. Yuan, J. Z. Leibo, and N. Jaques. Investesg: A multi-agent reinforcement learning benchmark for studying climate investment social dilemma, 2025. URL httpt:tt/////.....tt :tttt::::://iiv../// /arxiv. org/ brg/ 2411 09856.
M. Kearns and S. Singh. Bias-variance error bounds for temporal difference updates. N. Cesa- Bianchi and A. Goldman, editors, Proceedings of the Thirteenth Annual Conference on Computa- tional Learning Theory (COLT 2000), June 28 -Jly / 2000, Palo Alto, California, USA, pages 142- 147, Morgan Kaufmann, 2000. URL https: / /t/lbllp.... orr. wttp://///ee//ooeeeeeeeeeeeeeeoe conf colt /earnssSO.
D. P. Kingma and J. Ba. Adam: method for stochastic optimization, 2017. URL https:
/ //arxiv.org//e/ooe. org/ abs /1 1.112.66 6980.
A. Letcher, J. Foerster, D. Balduzzi, T. Rocktäschel, and S. Whiteson. Stable opponent shaping in differentiable games, 2021.
C. Lu, T. Willi, C. S. de Witt, and Foerster. Model-free opponent shaping, 2022.
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning, 2013. URL httpa : /atpiv..ivv/v/////////e/////aav/ org / 1.1 2. 5602.
10

---

Published as a conference paper at ICLR 2026
J. F. Nash. Equilibrium points in n-person games. Proceedings of the National Academy of Sciences of the United States of America, 36(1):48 49, 1950. doi: 10.1073/pnas.6.... 1.. URL httpa: // //www. pnas org. www.doioooooo 1ttp://///....1///..... 3....
1.48.
N. Nisan, T. Roughgarden, Éva Tardos, and V. Vazirani, editors. Algorithmic Game Theory. Cam- bridge University Press, New York, NY, USA. 2007. ISBN 978-0-521-87282-9,
H.-O. Pörtner, D. Roberts, M. Tignor, E. Poloczanska, K. Mintenbeck, A. Alegría, M. Craig, S. Langsdorf, S. Löschke, V. Möller, A. Okem, and B. Rama, editors. Climate Change 2022: Impacts, Adaptation and Vulnerability, Cambridge University Press, Cambridge, UK and New York, NY, USA, 2022. doi: 10. 00.1017/988100935555
URL htttps://doi.oot
9781009325844
A. Rapoport and A. Chammah, Prisoner's Dilemma: A Study in Conflict and Cooperation. University of Michigan Press, 1965.
T. Sandholm and R. Crites. Multiagent reinforcement learning in the iterated prisoner's dilemma. Bio Systems, 1766-2):177-166 1996.
Schulman, Wolski, P. Dhariwal, A. Radford, and Klimov. Proximal policy optimization algorithms, 2017.
J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation, 2018
L. Shapley. Stochastic games. Proceedings of the national academy of sciences, 39110):095--1100 1953.
von Neumann and O. Morgenstem. Theory of Games and Economic Behavior. Princeton University Press, Princeton, NJ, 1944.
X. Wang, H. Xu, Y. Zheng, and X. Zhan. Carbon market simulation with adaptive mechanism design. In Proceedings of the 31st International Joint Conference on Artificial Intelligence (IJCAI) Demo Track, 2024.
Willi, A. Letcher, J. Treutlein, and J. Foerster. Cola: Consistent learning with opponent-learning awareness, 2022.
R. Williams. Simple statistical gralient-following algorithms for connectionist reinforcmmenttt learning. Machine Learning, 8:229-256, 1992.
C. Yu, A. Velu, E. Vinitsky. J. Gao, Y. Wang. A. Bayen, and Wu. The surprising effertteeessssssssseeee.es of ppo in cooperative, multi-agent games, 2022. URL https: / /arxiv. 22ggabbs2203. 0190019995 955.
T. Zhang, A. Williams, R. Phade, S. Srinivasa, Zhang, Gupta, Y. Bengio, and S. Zheng. Ai for global climate cooperation: Modeling global climate negotiations. agreements, and long-ten term cooperation in rice-n. Social Science Research Network, 2022. doi: 10.48550/arXiv.2000 070004.
Zhao, C. Lu, R. B. Grosse, and J. N. Foerster. Proximal learning with opponent-learning awareness, 2022.
S. Zheng, A. Trott, S. Srinivasa, D. C. Parkes, and R. Socher. The ai economist: Optimal economic policy design via two-level deep reinforcement learning, 2021. URL https: / http///xxxvvrrr///// abs/2108 02755.
11

---

Published as a conference paper at ICLR 2026
APPENDIX
Social Dilemma Definition
13
Mathematical Derivations
14
B.1 Individual Case Analysis
14
B.2 Dependence on all Previous Mitigation Actions
15
B.3 Induction Argument Re-stated
17
B.4 All Selfish Gradients are Negative at Low Enough X
17
B.5 Some Selfish Gradients are Positive
17
B.6 Main Result
18
C On the Cooperative Bias of Advantage Alignment
19
D Experimental Details
20
D.1 PPO Policy Parameterization and Hyper- Parameters
20
D.2 Advantage Alignment Policy Parameterization and Hyper- Parameters
21
D.3 Baseline Scope and Rationale
22
Additional Figures
E.1 Ablation over the parameter
23
E,2 Additional Results in the Default InvestESG Configuration
24
E.3 Interpreting Neural Network Policies
25
12

---

Published as a conference paper at ICLR 2026
A SOCIAL DILEMMA DEFINITION
Definition 5 (Social Dilemma). Let ax be a Nash equilibrium of the game. A game is said to be social dilemma if it satisfies both of the following conditions:
1. Collective Inefficiency: The socially optimal outcome yields strictly higher total utility than the Nash equilibrium:
Σ (r((a))
Σ (ax)).
(17)
2. Individual Incentive to Defect: For each player i, there exists some joint strategy a such that deviating from the cooperative strategy is individually beneficial:
r ad))aaa) > (dc:
(18)
where a is a defection action and ac denotes the cooperative actions of all other players.
These two conditions capture the tension the heart social dilemmas: cooperation is globally optimal, but locally unstable due to unilateral incentives to defect (Capraro and Halpern, 2015).
However, this definition makes strong assumptions that may not hold in the real world (e.g. uniqueness of Nash).
13

---

Published as a conference paper at ICLR 2026
B MATHEMATICAL DERIVATIONS
In this section we prove the main result of this work, namely, that InvestESG is social dilemma for the correct choice the climate mitigation effectiveness parameter X. We begin by stating our assumptions about the environment:
Assumption 1. (Static Investor Portfolios). Investor holdings are time-invariant. Formally, Hy, = Hig Vt 2 0, E [M] j E [NT Thus investors never rebalance their capital allocations.
Assumption 2. (Single-Company Allocation). Each investor supplies capital exactly one firm.
Assumption 3. (No Historical Mitigation). Prior to the period of interest. firms have undertaken no mitigation expenditure: u²g = 4 < I, i E [M]
B. INDIVIDUAL CASE ANALYSIS
The capital of company is:
Ki+1 (1 pt) ++1
(19)
interii
1 = (1 (1 + 71
X,L:)
-
(20)
N
N
K
Ki (+t1iint
H K Σ
(21)
where X? is the capital of the j-th investor. In this argument will isolate the companies' behavior by assuming that the investors do not change their capital allocations over time, and that each investor invests in single company. This means that the money returned investors is the same the money investors put in at the current time-step. Giving us
Ki+1,imerim = K
(22)
Plugging this into the equation for Ki+1 gives us:
Kit (1 (1- (1) +1) X(I.)) Kt-
(23)
Here X, is the number of climate events time I, can go up if extreme heat, precipitation and drought all happen. The probability of cach of those events in X, depends on the mitigation up to time t, called Uy:
Het
P
+ Po
=
(24)
1 X.U,
Let's consider a company which has never done any mitigation in the past, so U,-1 = 0, but is considering doing at this time-step, it has an incentive to start mitigating. If this derivative is positive, then the company has an immediate, myopic incentive to take mitigation actions.
differentiate E E[K]+a] with respect to uę to investigate if
For every time-ster step t, company and scalar a, O, we have that the social marginal gradient is strictly greater than the private marginal gradient.
Lemma 1.
du Σ Ea,B K2H1
>
d du:
E Kitt
(25)
3
Proof.
d du
d duf
E (((11 u1) +1) xx,)) K
E
K
K
(26)
The only parts of this equation that depend on u; are the factor of (1 - ut) and through (1 (1- -X,L,)... Now for small values of climate risk events, approximate X, Bernoulli variable with
14

---

Published as a conference paper at ICLR 2026
probability P. =
Pt' hence the expectation above can be written
X, {0, the
algebraic trick used here comes from fa, y, 2)
= zyz
=
df /d = yz = fzz y, 2) xz giving us:
d
E
K 1+1
Ki+
dP:
K
IE
(27)
duf
wi
XeLi dut
And now for referring back to equation 24: dm?"
dr,
dP.
dU. d duj dut dU,
Σ Σ(1
(++UUU
Net
Po
K2 b+ Linterim
Σ 2(1+ XeU,)2
-aflet
(28)
Where the factor of K inneii comes from the fact that U, = Eser u²KK%_ And finally:
d
E u + +E
K
Aetlet
E Ki+1 dut
Ki+ Σ
(29)
X,I.,
(1+
This equation places lower bound on how incentivized selfish company is to invest in mitigation, does not include the long-term effects of mitigation all. If de is set high enough that this derivative positive, we do not have a social dilemma at all. Noticing that the equations the beginning of the section),
= /2+1/(1+ + pt) (from
replace:
d du.
[[Ki+]]
+ E 4 (1 X,I,) w1)11 +7) Σ Xepaot
(K
K 2+1
(30)
(1+
We can also trivially run the same steps for
Ki with j * i, i.e. computing the derivative of the return of agent w.r.t the mitigation of agent which will remove the first in the previous equation. The cumulative mitigation U, includes terms from all agents, so the contribution of the
du:
(1- X,L,) term the gradient remains:
for i + j: dut
K'
K1+1
K Kit1 (1 X, X,40(1 o
Σ
deMet (1+ X,U,)
(31)
We notice that this strictly non-negative, as expected (since all Kÿ are non-negative). All agents would want someone else mitigate for them, they get all the benefits of mitigation without any of the costs. Together with the non-negativity of Kj, we get the following inequality:
d duc
Vi
Σ
K EBKL11
E
E
K. K
(32)
duf
which falls straight out of equation 31 being strictly non-negative. This means that the "social derivative" always greater than the selfish derivative for mitigation actions ut. In particular, we also get the extra inequality:
d a,
d as dup
iADA E n), Vas 00
Σ
K2+1
Kitt
(33)
du.
which we will need later.
B.2 DEPENDENCE ALL PREVIOUS MITIGATION ACTIONS
Lemma 2. For every previous time- t - k. 0 < k < t, company i and scalar > 0, we have that the social gradient is strictly greater than the private gradient.
Ea
a
K
a.E
Kit
(34)
du
dui=k
Proof. Writing the main equation again for visibility:
(1- (i)) +1 XxI.)) Kt-
Kit1
(23)
15

---

Published as a conference paper at ICLR 2026
To compute the dependence of Ki+1 ui_ for general & 2 1, note that this enters in two ways in the equation above, first through the effect on P. which affects K2+ through (1 - -1) + 7) X,L,), and this will give us a factor like the second one in equation 29. The second way is through effects on K2, which were not present in the previous section:
Ki+1
dP:
+ (L- >) P.l.))
d dui=
E K -
B
(35)
duj
1- X,L; du
We have used the independence of X, and K2 to push in the expectation values. These two variables are independent if the past mitigation actions are kept constant, which is the case for us, since we are analyzing the case where all companies have never invested in mitigation (by construction). d. / k is computed before:
dP. dU, d
dut h dU,- I+
E(1+
Hel heUe
Hel
+ P6)
K Ki -I-I-Rkintriinn
Σ
[(++ XeU,)2
-ooPee
(36)
du h
We can now equation 35 to induce recurrence relation between the derivatives of all factors
E
k r+1
All put together, the recurrence relation is
d
11 (1 (1- (1,X)1))3(11 - nt)(1 + 7)
Aathat
E
Ki
E
Σ
duk
(1 + UeU
(1= 7)(1 +)) -P:Li)
d
Ki
(37)
du
And for the derivative with a different agent / + i:
d du
Ki+1 K4+2
Nepet
E
K!
=E
(1 X,T4((1 - X1 - (1) + +7)
(1 X,U))
k
+ (1 (1- + +1 P.L.))
(1-
d
E
K
(38)
dut
We now wish to get the equivalent of lemma for mitigations ui_ at any previous time-step. We use a proof by induction on the index & from the base case of k = 0 in equation 33, Let a; > be strictly positive real numbers:
du Σ aFE Kit1
d
Ki+1 K'+1
deHtet
Σ aFE [1 X,,,,)) -
Σ
(1 + XoU
+ Σ an (1 (1- (1) + +1 -P(ta)
d
K!
(39)
dut k
We now invoke the induction assumption for the term in the last line, which matches the induction case with af = au ((L w1)(1 + (1 - P.Li)). this is why we needed the assumption with a positive linear combination. The assumption gives us the following:
d
Σ au (1 (1-
uf)(1 (1-
+ )1 -PP-i)
E
K?
>
du
k
as (a-au)) (1 7) +) P.L.))
d
E
Ki
(40)
dut
But now we note that the elements in the first sum above are all positive, and so Kit
Σ E (1 - X1L.)(1 - X,L,)(1 - u1) + 7)
Σ
deplet (1 + XoU,)²
(Ki+1)²
Netlet (1)22
a,E
Σ
(41)
(1 (1-X, X1xx)1))2(1 - u1) + +7)
16

---

Published as a conference paper at ICLR 2026
This suffices to establish the desired inequality
d
du ai
Σ
Σa.B
Kits
a;E
Ki
(42)
dut_
Which as a special case (a: 1 for all 1) tells that the gradient of the social welfare with respect to the mitigation actions of any agent at any point in the past is higher than the selfish gradient. [
B.3 INDUCTION ARGUMENT RE-STATED
Theorem (Social-mitigation) For all > O, i
d duc
d E
E
Σ
Eke+1
Kitt
[ktt1]
(43)
dui_k
Proof. In order prove theorem 1, we need to prove the following equation for all i,t, k:
d
d
EanE
anE
K/+1
a;E
E
K Ki+l
(44)
dut k
duf
we proceed by induction on k using i 0 as base case:
Induction base case ls = 0. The base case is trivial from lemma 1.
Induction recursion. For the induction, we assume that equation 44 holds for all k' < k, and show that this implics that the equation true for This is what equations 40 through 41 are showing.
By setting aj = in the equation, we finish the proof.
B.4 ALL SELFISH GRADIENTS ARE NEGATIVE AT Low ENOUGH
Theorem 2 (Low mitigation effectiveness).
There exists Atow > 0 such that for all X < Niows we
have:
d dut_
d
E
K 1+1
0 and dut
Σ E
K < Vi,t
(45)
Proof. We proceed by simple recurrence to show that all gradients are negative at X, 0. We note that equation at A reduces to
[[K]+1]
E
K 2+1
(46)
du
m
thus this gradient is negative. Using equation 37, we see that the first term vanished in the d 0. giving us a term which only depends on duf
Ki
which we know negative:
d E Kitn E
-a--
P.L.))
d
E K
0.
(47)
du
du
Thus there exists Ae such that all the selfish gradients are negative, completing the proof.
B.5 SOME SELFISH GRADIENTS ARE POSITIVE Theorem (Self-interested mitigation). There exists such that:
d
Vt, 3k,
E k >
(48)
duf
17

---

Published as a conference paper at ICLR 2026
Proof.
Again from equation 30:
d
[[ ui
(K++1)² (1 - (,,,) ut (1 + 7) E
defeet
Kitk
(1+
(49)
du:
We evaluate the gradient at the U2 0 policy, hence can isolate the de where the sign flips:
(Ki+1)²
E[K 1144
Asign dip
Σ Not
(50)
(1 - x1) +
which implies that at À sufficiently high, least some selfish gradients become positive, and the theorem is proved.
B.6 MAIN RESULT
We can now state the main social dilemma theorem, which says that there is a À where the individual gradients are strictly negative, but at least some social gradients are positive.
Corollary 1 (InvestESG is a Social Dilemma). There exists some mitigation effectiveness parameter such that
Vi,ik, dui_k
[[ < and Vt Bi,k
Σ E E[[H+1
> 0.
(51)
dul k
Proof. From the intermediate value theorem and theorems 2 and 3, we conclude that there ex- ists Aeritical for which St, i, k. du -- Ki+1] O, and from continuity of d - with re- du;
spect to À infer that there exists d E [[]++1] < O. Now theorem applied Auiiical tells 3t, i, k it, h du
0 such that for Aritical À Aeriical
have du;_ B E K
0, and again by continuity this quantity remains positive at some smaller Aeritical E. Thus Acnical min(c.e") ensures that all individual gradients are negative while there exists a social gradient which is positive.
18

---

Published as a conference paper at ICLR 2026
C ON THE COOPERATIVE BIAS OF ADVANTAGE ALIGNMENT
Advantage Alignment has a bias towards promoting cooperation between agents when the advantages estimated using generalized advantage estimation (GAE) (Schulman al., 2018). The main insight is that TD-1 errors have a positive bias for on-policy trajectories because the critic lags behind the actor; advantages are systematically underestimated, producing a cooperation bias in the algorithm. The advantage alignment updated advantages are:
(s,, ar) - ('(, an) + B
Σ S#1
C-tk A' an) A (se, au)
The term Exse z²= A' ax) acts multiplicative factor for the opponent advantages. If we let b' E PnPi Ex A ax)]. we can split the expression into its biased and unbiased parts:
binsed par (8t; ar)
(t) +0x 37 Σ As
(52)
5#1
A n
unbinsed part (st, ar) = By
Σ jzi
Cb- 11 A'kk k<C
A' (Sc. as).
(53)
It is this biased part of the new advantages which produce the cooperation bias. In the extreme case where b B- the learning process collapses to simply doing summed-reward training.
For advantage functions (s a) which perfectly evaluate the behavior of the policies Ty. the terms above have b' = yielding no collaborative bias for advantage alignment. However, Generalized Advantage Estimation computes the advantages as:
oY = + Vo(st+1) Vo(a.),
(54)
00
A. AGAE(A))
= Σ (T
(55)
Where the value networks V6 are trained to evaluate policies 7e which are themselves learning and improving. As improves, the value networks perpetually lag behind, computing the value of slightly time-delayed policies. This implies that (st) will on average underestimate the next-step rewards Kearns and Singh (2000), and this will give positive bias to the TD-errors on-policy:
E - [6
(56)
T
A
GAB(A,))
(57)
8' P
1- A.CAAE(A)))
(Sks ap))
> 0.
(58)
This bias towards cooperation only manifests at the beginning of training, before the critic has fully caught up with the actor. At convergence of the actor, the critic does catch up, and the TD-errors become unbiased, removing the cooperative bias.
19

---

Published as a conference paper at ICLR 2026
D EXPERIMENTAL DETAILS
Throughout our experiments, we trained using the Adam optimizer (Kingma and Ba, 2017) and trained for 70 million environment steps (approximately 20000 gradient steps), anything beyond that resulted in lower performance. Our experiments ran in approximately 4 hours on a NVIDIA L40s gpu.
D.1 PPO POLICY PARAMETERIZATION AND HYPER--PAAAMEE
As in the InvestESG paper (Hou et al., 2025), size 256 neurons and ReLU activations to parameterize the mean of a multivariate normal distribu- tion. We use additional parameter to parameterize the log standard deviation. For the critic, use the same architecture (two-layer MLP, ReLU activations, with hidden size 256) to estimate the value directly. We swept hyper-parameters to find the best for maximizing market total wealth. For all PPO experiments we used:
two-layer fully-connected MLP with hidden
Hyperparameter
Value
Algorithm
IPPO
Total Environment Steps
70M
Number of Environments
64
Episode Length
100
Discount Factor (?)
0.99
GAE X
0.95
Policy Learning Rate
le-4
Value Function Learning Rate
le-4
Entropy Coefficient
0.05
Clip Ratio (c)
0.2
Value Function Clip Range
10
Number of PPO Epochs
Number of Minibatches
20
Gradient Clipping
10.0
MLP Hidden Size
64
MLP Hidden Layers
2
Self-Play
False
Fixed Random Seed
False
Table 1: Key PPO hyperparameters used for training in the InvestESG environment.
In the original InvestESG paper, the stochasticity of the environment is fixed with single seed. We ran experiments in the more challenging stochastic setting, which accurately reflects some of the nuances of the climate problem.
20

---

Published as a conference paper at ICLR 2026
D.2 ADVANTAGE ALIGNMENT POLICY PARAMETERIZATION AND HYPER- PRREMPPRRAMEEER
As in theprevious section, we use two-layer fully-connected MLP with hidden size of 256 neurons and ReLU activations parameterize the mean of multivariate normal distribution. We use an additional parameter to parameterize the log standard deviation. For the critic, we use the same architecture (two-layer MLP, ReLU activations, with hidden size 256) estimate the value directly. We swept over Prantage-Aligmmeen specific fixed) to find the best for maximizing market total wealth. For all Advantage Alignment experiments we used:
(keeping the PPO hyper-parameter
Hyperparameter
Value
Algorithm
AdAlign
Total Environment Steps
70M
Number of Environments
64
Episode Length
100
Discount Factor (*)
0.99
GAE
0.95
Policy Learning Rate
le-4
Value Function Learning Rate
le-4
Entropy Coefficient
0.05
Clip Ratio (e)
0.2
Value Function Clip Range
10
Number of PPO Epochs
1
Number of Minibatches
20
Gradient Clipping
10.0
Use RNN
Falsc
MLP Hidden Layers
2
Self-Play
True
Advantage Alignment /
0.2
Advantage Alignment y
0.9
Fixed Random Seed
False
Table 2: Key Advantage Alignment hyperparameters used for training in the InvestESG environment.
As mentioned before, using self- play greatly improves the stability of Advantage Alignment runs across seeds, therefore we use it for all Advantage Alignment experiments.
21

---

Published as a conference paper at ICLR 2026
D.3 BASELINE SCOPE AND RATIONALE
We focus on methods that (i) handle continuous actions and high-dimenniionll observations, (ii) train under realistic compute budgets, and (iii) have been used in practice. This leads IPPO and MAPPO as the main baselines, and excludes several prior opponent-shaping (OS) methods for the full InvestESG experiments.
IPPO exposes non-stationarity and tredit-assigmment challenges with decentralized critics and is strong, commonly used baseline in general-sum MARL. MAPPO follows CTDE: actors use local observations at train/test time while a centralized critic conditions on global context during training to reduce variance and improve credit assignment. In our setting, MAPPO offers a scalable, fair comparator that addresses the pathologies of IPPO without requiring execution-time communication.
Method
Continuous actions
Practical runtime
Used scale
LOLA
M-FOS
BRS
X
LOQA
X
AdAlign
LOLA requires higher-order differentiation and has been found suboptimal even on simple discrete games (see LOQA Aghajohari al. (2024b)), and it is not designed for continuous control. M-FOS has similar limitations to LOLA with reported underperformance on small discrete settings. BRS is computationally prohibitive; reports indicate the order of ~48 A100 GPU-hours per seed on Coin Game, making InvestESG-scale sweeps impractical Aghajohari et al. (2024a). LOQA relies normalized action probabilities, which makes it incompatible with continuous-action parameterizations. Advantage Aligmment is policy-gradien based and action-agnosee and has been deployed in nigher-dimession settings where prior OS methods are typically not compared for similar scalability reasons.
22

---

Published as a conference paper at ICLR 2026
E ADDITIONAL FIGURES
E.
ABLATION OVER THE a PARAMETER
Mitigation Investment
Final
300
400
0.5
3000
100
2003
0.3
1000
0.1
0.0
Status ESG
ESG
Status E5G
DSG
Miúgation
15000
1200
12500
1000
820
0.3
7500
0.2
5000
230
0.0
Stadus Quo ESG 0.5 ESG
ESG
10
stadus ESS 0.5 E5G
850
10
Stat
ES6 05
ESS
856
Market Total
Mitigation Investment
Climate
0.5
1000
0.3
750
5000
250
0.6
Status
ESS
ESG
ESG
Market Total Wealth
Final Midigation Investment
Climate Risk
1500
0.5
15000
100
1000
0.3
500
0.0
ESC
ESG
E55
Status 0.5
150
D5C 0.5
Figure 5: Comparison of final environment metrics at different a (introduced in equation 9) values, for 10 seeds of PPO agents. We try values of {1, 50, 70, 100} for the parameter a. The choice of 70 is the most sensible one. as there clear differences between all policies with different ESG incentives. The result, alheit similar, is less apparent with choice of 100.
23

---

Published as a conference paper at ICLR 2026
E.2
ADDITIONAL RESULTS IN THE DEFAULT INVESTESG CONFIGURATION
Schaling Diagram (MTW): Target vs. Others Composition
Defector
2500
Conzerator
3200
Moho5 Rita Weath Dag)
2300
1500
1000
302
Coopersting
Schelling diagram looking the market total wealth with a cooperator (green) a defector (red) varying the number other cooperating policies on the x-axis, Cooperation here defined as in the original InvestESG paper, where a company spends 0.5% of their capital on mitigation.
Figure 6:
Figure 6 empirically shows that the cooperative action profile in the original InvestESG paper (0.5% of capital spent on mitigation) not well calibrated. This is due to the fact that the market total wealth achieved by the cooperative action profile (+= 3500) is lower than the one achieved by PPO agents (re 4000) in Figure 5. This contradicts both definition and the intuitive construction that we provide of social dilemma using the price of anarchy in definition 2.
24

---

Published as a conference paper at ICLR 2026
E.3
INTERPRETING NEURAL NETWORK POLICIES
comaeny Comten
2300
a000
1753
Com vary
5000
000
700
Company Miagaton Irvastmento Cepp Times
Company Captials Time
Inaedrrent Matta
Compane
Con 2an0
12:00
Compune
Com sam
Compune
600
200
19AM
240
Figure 7: Policy dynamics in the ESG = 0 social dilemma. Top: Advantage Alignment (AA); bottom: PPO baseline. (i) Mitigation time: AA invests only climate-riss risk peaks and then returns zero, whereas PPO agents invest erratically throughout. (ii) Company capital: AA keeps wealth roughly uniform across firms, while PPO gradually concentrates it. (iii) Final investment matrix: AA yields an almost symmetric allocation, whereas PPO shows fragmented, concentrated investments.
25


Reviews:
[{"review_id": "lloTwrZZSU", "reviewer": "Reviewer_mc7N", "summary": "The authors develop a novel MARL-environment, Invest-ESG, that under certain parameterizations assumes the form of a social dillema. Furthermore, the authors employ Advantage Alignment (AA) for opponent shaping and compare to MAPPO and IPPO baselines. Finally, they provide a theoretical explanation of why yields superior equilibria compared to baselines.", "questions": "Q1) Could you include results using AA without disabling greenwashing.\n\nQ2) Have you explored utilizing heterogenous lambda values as certain industrial sectors could impact the likelihood of certain climate hazards? e.g. extractive industries and manufacturing.\n\nQ3) Are there existing econometric models of impact investing that are to some degree comparable with the results seen here?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Originality:\nThe paper introduces a novel MARL environment that highlights a critical link in broader climate-economic space, namely impact investing and greenwashing risks. Furthermore, the authors make use of SOTA learning algorithms.\n\n- Quality:\n\nThe authors are theoretically rigorous in their analysis and include an appendix proving that InvestESG is a social dillema for certain values of lambda along with ablation studies.\n\n- Significance:\n\nSignificance largely lies in highlighting the scalability of AA even in a high-dimensional MARL context. Authors provide insight into learned policies through gini coefficient analysis, final mitigation investments, market wealth and climate risk.\n\nClarity:\n\nPaper is clear and well structured.", "weaknesses": "-The real world impact is overstated. the problem with building international climate agreements is that cooperation is difficult to achieve, using an algorithm that is biased towards cooperative policies doesn’t really capture that phenomena. Missing some connection to actual climate-economic literature about the dynamics of impact investing.\n\n-Theoretical results only valid under strong, unrealistic assumptions.\n\n-Interesting components (greenwashing, resilience investments) of environment disabled.\n\nI will reconsider my score based on the answers to the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UPAK76lroq", "reviewer": "Reviewer_b9EZ", "summary": "This paper rigorously analyzes the InvestESG environment. They first formally identify the parameter $\\alpha$ required to establish a true intertemporal social dilemma. The authors then apply Advantage Alignment, a scalable opponent-shaping algorithm, to this calibrated environment. This method effectively steers agents toward cooperative, high-welfare equilibria, outperforming standard MARL baselines even without external ESG incentives. The work is supported by a theoretical argument explaining why this opponent-shaping approach is inherently biased toward finding such socially beneficial outcomes.", "questions": "Your theoretical argument in Section 6 that AA's success stems from a \"cooperative bias\" induced by GAE critic lag is a central and intriguing claim. However, this mechanism is never empirically demonstrated. Could you provide data from your training runs to substantiate this? For example, could you plot the on-policy mean of the advantage estimates ($b^i$ in your derivation) over time? Seeing this term be positive and decay as the critic converges would provide strong empirical evidence that this is the active mechanism.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper's primary strength lies in its formal analysis of the InvestESG simulation. Instead of taking the environment at face value, the authors mathematically derive the precise conditions under which it functions as a true social dilemma.\n\n- The successful application of Advantage Alignment to this complex high-dimensional economic simulation. It demonstrates a scalable method for finding cooperative high-welfare solutions where standard MARL baselines fail.", "weaknesses": "- The paper justifies excluding other OS methods like LOLA or BRS on the grounds of scalability. While reasonable, this means AA is only compared against non-shaping methods (IPPO/MAPPO). It is unclear if AA superior because it's an OS method, or because it's a better OS method. Comparing AA to at least one other OS method on a scaled-down version of the $\\alpha$-InvestESG environment would be helpful to make a stronger claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YskwqcRlj5", "reviewer": "Reviewer_TMZo", "summary": "The paper applies Advantage Alignment (an opponent shaping algorithm) on a varied version of InvestESG environment, a realistic simulation of corporate and investor interactions under climate risk, and compared to baseline algorithms such as IPPO and MAPPO.\nIts major contributions include:\n\n1. Formally prove that the parameter $\\alpha$ (the climate responsiveness to mitigation) is critical for making the game a social dilemma in the stochastic setting.\n1. Prove AdAlign is more effective in finding social welfare maximizing strategies compared to baseline algorithms both theoretically and empirically.", "questions": "**Real-world implication**: The merit of InvestESG is rooted from its depiction of company-investor interactions in the real world. With the theoretical calculation that $\\alpha$=70 leads to a full social dilemma, which makes mitigation much more potent. Can you justify this magnitude in terms of real-world ranges?\n\n**Presentations**: The paper spends Section 2.1, 2.2, 2.3 on formulating Markov Games, RL, and Social Dilemma. The application to Markov games is relatively standard. For better presentation, the paper should focus on its novelty point.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. **Clear analysis and diagnosis of InvestESG benchmark**: The paper formalizes the conditions when InvestESG is a dilemma. Then it validates the predicted 𝛼 threshold empirically: the single-firm/investor sweep exhibiting a sharp change near 𝛼≈30 and full game behavior at 𝛼=70. The proof is rigorous and the empirical implementation is well-executed. This is very useful to the community who would like to utilize this benchmark for policy analysis and policy making.\n1. **Comparison between AdAlign and baselines**: In Section 5.1, the paper compares the final welfare achieved by AdAlign with the result achieved by IPPO and MAPPO, and also gives their interpretation about how opponent shaping alters investment behaviors. The analysis also include inequality analysis. In Section C.3 the paper also addresses the reason why AdAlign is the most applicable option among all the opponent shaping algorithms.", "weaknesses": "**Limited novelty**: The paper focuses on one *existing* benchmark (with some modifications) for deep analysis, and applies an *existing* opponent shaping algorithm on this benchmark. However, the results of original AdAlign paper show that this method has proved to be effective in maximizing social welfare compared to PPO baseline on other benchmarks. Applying the same method to a variation of InvestESG might not considered to be a *fundamental* innovation.\n\n**Asymmetry in comparisons**: The paper claims that AdAlign benefits from self-play (one set of parameters for the company and another set of parameters for investor players) while PPO agents don't. However, it muddies conclusions about algorithmic superiority versus training-regime effects. More ablations (e.g., AdAlign without self-play; PPO with tuned self-play variants) would help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Iww1hiGx1Q", "reviewer": "Reviewer_EiHm", "summary": "This paper studies the InvestESG model (ICLR 2025) and provides both theoretical and empirical evidence that the Advantage Alignment Algorithm (ICLR 2025) - an opponent shaping algorithm - scales better and finds more efficient equilibria than common baselines PPO, and MAPPO. More formally, the paper equips the original InvestESG model with a parameter that it terms 'climate responsiveness parameter' and shows that a simplified version of this augmented InvestESG model exhibits characteristics of a social dilemma (unlike the general case, as claimed in the original InvestESG paper).", "questions": "In addition to discussing the weaknesses above, I was slightly confused about the claim that the original InvestESG model is not a social dilemma for its original parameter configuration. It would be great if the authors could clarify this statement. In particular, does this rectify an inaccurate claim in the original paper where the contrary is claimed?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Originality: the paper applies an opponent shaping method to InvestESG showing that this algorithm outperforms baselines in terms of scalability and equilibrium selection. Also, if I understand correctly, the paper rectifies the claim that the original InvestESG is a social dilemma for all parameter configurations. Thus, originality comes essentially from the synthesis of two recent ideas.\n\nQuality and Clarity: the paper is generally clearly written and rigorous with proofs being correct to the extent that I could verify. The experimental results contain enough detail to be reproducible.\n\nSignificance: the paper contributes to the literature about using ML to tackle social problems, and in this case, climate change.", "weaknesses": "The main weaknesses in my opinion are the following:\n\n- While the paper is clear in its presentation, it does not discuss enough its most important modelling decisions and assumptions, i.e., whether the model of a stochastic game is indeed adequate to capture climate change and whether the climate responsiveness parameter is rich enough to capture something valuable. In particular, a value of \\(\\alpha=70\\) seems to correspond to a rather unrealistic setting: \\(\\lambda\\) is calibrated to reflect the 1.3 trillion commitment and scaling this by 70 seems to result in a setting of only theoretical interest. Whithout aiming to re-review the original InvestESG model, I think that this paper does not help to lift the limitations of the original model, but rather introduces some more need for discussion based on the points that I mention above. \n\n- The paper claims as its main contribution that Advantage Alignment outperforms (various variants of) PPO in terms of equilibrium efficiency and scalability. In terms of practical importance, I am not sure what exactly we learn from this in the particular climate change setting. Shall we expect that companies will be prescribed to use the AA algorithm and, thus, achieve better outcomes in (toy model) of battling climate change? Why is opponent shaping relevant in this model and why should a company use (or a government enforce/incentivise its use). Since the paper's intented contribution seems to be the climate change mitigation, I found that the practical relevance of modeling and results was lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)