Exploring Equity of Climate Policies using Multi-Agent Multi-Objective Reinforcement Learning
Palok Biswas [,][∗], Zuzanna Osika [,][∗], Isidoro Tamassia [∗], Adit Whorra [∗], Jazmin Zatarain-Salazar, Jan Kwakkel, Frans A. Oliehoek and Pradeep K. Murukannaiah
1 Delft University of Technology, The Netherlands { p.biswas, z.osika, j.zatarainsalazar, j.h.kwakkel, f.a.oliehoek, p.k.murukannaiah } @tudelft.nl isidorotamassia@gmail.com, aditwhorra@gmail.com ∗ Equal contribution
Abstract
Addressing climate change requires coordinated policy efforts of nations worldwide. These efforts are informed by scientific reports, which rely in part on Integrated Assessment Models (IAMs), prominent tools used to assessthe economic impacts of climate policies. However, traditional IAMs optimize policies based on a single objective, limiting their ability to capture the trade-offs among economic growth, temperature goals, and climate justice.
As a result, policy recommendations have been criticized for perpetuating inequalities, fueling disagreements during policy negotiations. We introduce JUSTICE, the first framework integrating IAM with Multi-Objective Multi-Agent Reinforcement Learning (MOMARL). By incorporating multiple objectives, JUSTICE generates policy recommendationsthat shed light on equity while balancing climate and economic goals. Further, using multiple agents can provide a realistic representation of the interactions among the diverse policy actors. We identify equitable Pareto-optimal policies using our framework, which facilitates deliberative decision-making by presenting policymakers with the inherent trade-offs in climate and economic policy. 1 Introduction
Climate change poses a significant global threat, disproportionately affecting marginalized communities. The extent of thisthreat remains uncertain due to the complex interplay between climate and socioeconomic systems. This uncertainty underscores a central challenge of climate justice: the fair allocation of burdens and benefitsthrough the implementation of climate policies. The complexity in assessing the implications of policy measures leadsto contentious international negotiations, as seen in the Conference of the Parties (COP). Asymmetric impacts, divergent responsibilities, and differing national priorities often resultin impasses and heated disagreements among policymakers. Climate change isthus characterized as a wicked problem in the policy domain, necessitating interventionsthat seek to balance the diverse ethical preferences of stakeholders. The Intergovernmental Panel on Climate Change (IPCC) is a key player in the global climate change discourse, and its assessment reports serve asthe primary source of scientific information for international climate negotiations, such asthe COP. These technical reports, crafted by experts from various disciplines, including climate science and economics, provide evidence-based guidance to support effective climate action through the use of Integrated Assessment Models (IAMs). IAMs integrate socioeconomic, climate and technological processes into a single framework to assess global mitigation pathways and simulate future socio-economic and climate scenarios. Although IAMs are influential tools, they are criticized for suggesting inequitable mitigation policiesthat disadvantage developing countries. These models often simplify the complexity of climate change policiesto a single objective, which is ethically problematic, as it favours dominant stakeholders and neglects non-economic metrics, such astemperature, biodiversity, and human mortality. Recent studies seek to addressthese shortcomings by enhancing IAMs with multiple objectives, or by incorporating multi-agent approaches, such as multi-regional studies. However, the multi-objective studiestreat the world as a single agent, which overlooksthe interests of various stakeholders. Conversely, the multi-agent approach focuses on a single objective, neglecting the diverse preferences of agents during negotiations. Currently, no single modeling framework simultaneously addresses both the multiplicity of objectives and agents, thus overlooking the complexities of the real-world. The exclusion of either multiple objectives or multiple agents limitsthe applicability of IAMs. Our contribution fillsthis gap by developing J US TICE, a Multi-Objective Multi-Agent Reinforcement Learning (MOMARL) IAM framework that emulates real-world negotiations and discovers equitable policy options. MOMARL is a powerful framework for complex decision
1 making that requires balancing conflicting objectives and coordinating independent decisions over time. In particular, it is a crucial tool for addressing problemsthatinvolve sequential decisions. By extending reinforcement learning (RL) to accommodate multiple agents and multiple objectives, MOMARL enablestrade-off management through vectorial rewards, where each component reflects performance on a specific objective. Despite the prevalence of societal problems involving multiple stakeholders with diverse preferences, the field of MOMARL remains underexplored. Simplifications such as hard-coding trade-offs or centralizing decisions limit practical applicability.
Importantly, existing works are limited to theoretical examples or simple (grid-like) environments. Our contribution istwofold. For the IAM community, JUSTICE introduces a multi-objective, multi-agent framework of a climate-economy model, providing atool that can inform IPCC’s synthesis reports. Such a framework can enable the design of equitable policiesthat account for regional disparities while balancing multiple aspects of climate change, including economic and environmental outcomes. For the MOMARL community, JUSTICE offers a well-designed, open-source, implementation using the MOMALand API, enabling seamless integration with any RL algorithm. This provides a real-world testbed for evaluating and benchmarking future algorithms, supporting the development of robust and societal applications of reinforcement learning methods. 2 Background
Our interdisciplinary work is based on two foundational areas: climate modelling and reinforcement learning. 2.1 Integrated Assessment Models IAMs offer a holistic approach to inform policy decisions on climate change. These models integrate socioeconomic, technological, and biogeochemical variablesto representinteractions between simplified social and climate components. IAMs primarily inform climate mitigation policies and are frequently used for policy evaluation and optimization. Originating from William Nordhaus’s DICE model, which combined economic and climate models for global cost-benefit analyses of climate policies, IAMs analyze climate policy’s impacts on both the economy and environment. DICE is a foundational framework, earning Nordhaus a Nobel Prize and being utilized by the US government to calculate the social cost of carbon, which measuresthe societal benefit of reducing CO 2 emissions. DICE has inspired numerous IAMs and remains a valuable tool for assessing climate policies. A popular regional variation of DICE, known as RICE, encompasses 12 regions, and its recent and enhanced version called RICE50+ expandsto 57 regions for greater regional resolution. IAMs can be broadly categorized into two types: highly aggregated Cost-Benefit models and detailed process-based models. Cost-Benefit
IAMs (CB-IAMs) are optimization modelsthat identify nearterm emission reduction pathwaysto maximize long-term benefits by considering high-level economic and climate interactions. DICE/RICE IAMs fall under this CB-optimization category. In contrast, Process-Based IAMs (PB-IAMs) are simulation modelsthat offer detailed economic representations across multiple sectorsto analyze the impacts of specific policies on economic, social, and environmental factors, with an emphasis on sector-specific environmental impacts, such asthose in energy systems. CB-IAMs are employed for global mitigation pathways, macroeconomic assessments of mitigation strategies, and strategic interactions reported by the IPCC’s Working Group III (WGIII). Notably, the macroeconomic components of CB-IAMs form the foundation of many process-based models. The global mitigation pathways found by CB-IAMs can also be used to inform the simulation of PB-IAMs.
The DICE/RICE family, despite its simplicity, has significantly influenced IPCC WGIII mitigation assessments in various reports, including the recent sixth assessment report, particularly in Chapters 3 and 14, which focus on Mitigation Pathways and International Cooperation, respectively. 2.2 Multi-Objective Multi-Agent RL
We formalize a MOMARL problem as a multi-objective multi-agent Markov decision process (MOMAMDP) with team reward, defined as atuple � S, A, F, R �, comprising N ≥ 2 agents and d ≥ 2 objectives:
- Sisthe state space. - A = A 1 × · · · × AN isthe set of joint actions, where An denotesthe action set of agent n . - F : S × A → ∆( S ) isthe probabilistic transition function, mapping each state–joint action pair to a distribution over next states. - R = R 1 × · · · × Rn representsthe reward functions, where Rn : S × A × S → R [d] isthe vector-valued reward function for agent n across d objectives. Agents optimize policies π n to maximize expected discounted returns:
where π = isthe joint vector policy of the agents acting in the environment, γ isthe discount factor, and Rn isthe vectorial reward obtained by agent n at timestep t for the joint action at ∈A at state st ∈S . Solution Set Concept The value function v n [π] [∈] [R] [d] [ offers] a partial ordering over policies, as it is a vector. Identifying the optimal policy requires a utility function u n : R [d] → R to capture agents’ preferences by mapping vectorsto scalars. The Pareto set, assuming the utility function is monotonically increasing, uses Pareto dominance ( ≻ P ), where a vector dominates another if it is at least equal in all objectives and strictly better in one. When agents share ateam reward
�
v n [π] [=][ E]
∞ � γ [t] Rn | π � t =0, Pareto dominance applies di-] rectly. For a set of policies Π, the Pareto set P ( Π ) includes all undominated policies:
P ( Π ) = { π ∈ Π | ∄ π [′] ∈ Π : v [π] [′] ≻ P v [π] }. The Pareto front F ( P ) contains value vectors for Paretooptimal policies in P ( Π ). When the utility function is a positively weighted linear sum, the solution set formsthe convex hull of v [π]. 3 The JUSTICE Model
The JUSTICE IAM is a modular and efficient framework created to assess various modeling assumptions related to economic growth, damage functions, abatement costs, and social welfare. It allows experimentation with different uncertainties withoutincurring high computational costs. We provide an outline of the model and its MOMARL formulation. 3.1 High-Level Outline
JUSTICE incorporatesthe economy, damage, and abatement modules from the RICE50+ model and integratesthem with the FAIR climate model, facilitating probabilistic assessments of climate policies across various climate sensitivity scenarios. FAIR is a simplified emulator of complex climate models grounded in climate physicsthat accurately reproduces historical climate data and is utilized by the IPCC. Unlike other IAMs, JUSTICE identifies Paretooptimal policies in a multi-objective framework, assessing their robustness across various socioeconomic and climate uncertainties and considering the distributional impacts of the Pareto-optimal policies. JUSTICE operates on a yearly resolution and covers 57 independent regions. JUSTICE utilizesthe Shared Socioeconomic Pathway (SSP) dataset for exogenous regional economic growth, carbon intensity, and population projections, and the Representative Concentration Pathway (RCP) datasetinforms emission trajectories for other greenhouse gases and aerosols. The SSPs and RCPs provide a consistent framework for integrating socioeconomic and climate research, representing various future scenariosthrough different socioeconomic narratives and climate forcings. The exogenous SSP data is denoted with a superscript ( [∗] ) in the equations. Figure 1 provides an overview of the JUSTICE IAM. The economy sub-model employsthe neoclassical Cobb-Douglas production function, as in the DICE IAM, to compute economic growth using exogenous labour and total factor productivity data along with capital stock. The resulting economic output feedsthe Emissions submodel, which calculates CO 2 emissions at each timestep based on output and carbon intensity, defined asthe fossil fuel share in energy production. The CO 2 emissions from various regions are inputinto the FAIR climate model, which
2 ~~A~~ ppendix.pdf
aggregatesthem to calculate global mean surface temperature, expressed in °C above pre-industrial levels. This global temperature isthen downscaled to regional temperature rise using a data-driven statistical downscaler. The downscaled regional temperature is used by the Damage Function to compute the fraction of economic output damaged by temperature increases in every region. Additionally, the Emissions sub-model allows for CO 2 mitigation, with associated costs calculated by the Abatement module, based on the emission control rate chosen by agents (or regions) at each timestep. Detailed descriptions and equations for each model component are available in Appendix B . Figure 1: Overview of JUSTICE . The main outputs of the model have been highlighted with red arrows. 3.2 Multi-Agent Multi-Objective Formulation
We model JUSTICE as a MOMAMDP with team reward as specified below. Agents We model 12 agents, each representing a macroregion that groups countries following RICE region specification. A map of these macro regions with regional mapping table is provided in Appendix A 2 .
While JUSTICE can generate data for 57 entities, modelling a MOMARL problem with 57 regions is computationally expensive and complicates analysis. Actions Each agent’s action is atwo-dimensional discrete vector, where the actions of agent n are:
- Emission Control Rate (ECR) ∈{ 0 , 0 . 1 , 0 . 2 , . . ., 1 . 0 },
- Savings Rate (SR) ∈{ 0 , 0 . 1 , 0 . 2 , . . ., 1 . 0 } . ECR representsthe percentage of emissions removed from the atmosphere relative to the baseline SSP emissions. It is used in the Emissions module to assess emissions reduction, with the cost of this reduction being calculated in the Abatement module. SR determinesthe level of investment, indicating the amount of capital saved and reinvested into the economy, which in turn influencesthe economic output of each agent (a higher savings rate leadsto higher economic output). Each agent undertakesthese actions annually. Observations The agents can observe their local outcomes, the global temperature, and the mitigation rates set by all the agents at the previoustimestep. For n -th agent at timestep t, the observation and their ranges are as follows:
Ω t,n = Y t,n [GROSS] · ω t,n , Ω t,n ∈ isthe gross economic output and ω t,n isthe damage fraction for agent n at timestep t and is calculated in the Damage module of JUSTICE . Λ t,n = ζ t,n · ε [∗] t,n [·] � AC 21 t,n
Rewards The rewards are modelled asteam rewards, which assume collaboration among agents, where all agents receive the same return vector for executing the policy, reflecting their shared goal of improving global outcomes.
This setup can be easily adjusted to individual rewards if needed. In our approach, each agent receives atwo-dimensional vector of continuous rewards based on the following metrics:
- Inverse Global Temperature (IGT):
IGT t,n = 1 /GMT t , IGT t,n ∈ . - Global Economic Output (GEO):
1 t,n
2 ECR t,n [+] [AC] 5 [t][,][n]
[t] [,][n] ECR
5 t,n
, �
Λ t,n ∈ = Y t,n [GROSS] − Ω t,n − Λ t,n , Y t,n [NET] ∈ isthe net economic output of agent n at timestep tin trillion USD adjusted to 2005 PPP and is calculated in the Economy module of JUSTICE . ε t,n = CI [∗] t,n [·][ Y] t,n [ GROSS] · + AFOLU [∗] t,n [,]
ε t,n ∈ t,n isthe carbon intensity (exogenous SSP data), AFOLU [∗] t,n [isthe Agriculture,] Forestry, and Other Land Uses emissions (exogenous SSP data) of agent n at timestep t and is calculated in the Emissions module of JUSTICE .
GEO t,n =
N � Y t,n [NET] , GEO t,n ∈ includes additional details. 4.1 Performance Indicators
Evaluating and comparing solution sets in MOMARL is more complex than in single-agent, single-objective RL due to the lack of inherent ordering of solution sets and intertwined performance across objectives. This added complexity leadsto varied evaluation methods in MOMARL. To study convergence, we use two most commonly used approaches from both MORL and MARL domains for their suitability in MOMARL settings and to study the equality of distribution between agents we use GINI index:
GMT t = FAIR
ε t− 1 ,n
�� n∈N
, GMT t ∈ ,
where VECR t , n is a vector of emission control rates adopted by the agents at the previoustimestep that agent n observes at timestep t . Hypervolume ( ↑ ) representsthe region or (hyper-)volume between the points in the solution set and a reference point. The reference pointindicatesthe lower bound for each objective.
A solution set can be assessed by comparing its hypervolume with that of competing algorithms or the true Pareto front, if known. Expected utility ( ↑ ) When the decision-maker’s utility function u is linear, the expected utility (EU) metric can be used to represent the expected utility over a distribution of reward weights W . GINI Index ( ↓ ) The GINI index is a measure of inequality that quantifies disparities among agents based on a specific metric, with 0 indicating perfect equality and 1 indicating maximal inequality; we employ Concept-1 GINI by Milanovic to assess international inequality, where each agent represents a region and the index reflects whether their metrics are converging. 4.2 RICE50+ Comparison
We compare our results with the default RICE50+ model, as outlined in Section 2.1. This model consists of 57 regions and integrates optimization directly within the global climate simulation. The optimization follows a single-objective approach, employing asocial welfare function asthe global objective and using a single representative agent to optimize welfare, rather than a multi-agent multi-objective framework. The results presented in the next section are obtained using standard IAM methods, specifically non-linear programming within the General Algebraic Modeling Language (GAMS) for policy optimization. Unlike our approach, RICE50+ does not perform inter-temporal optimization at each timestep. Instead, it optimizes over the entire time horizon by using asocial welfare function as a proxy for consumption per capita. Since consumption per capita is directly derived from net economic output, we extract the net economic output from RICE50+ to compare with our objectives. 5 Results
The objectives of our experiments are to verify the convergence of JUSTICE MOMARL, compare the solutions between JUSTICE and the RICE50+ model, and analyze key solutions from the Pareto set. We present results in line with these objectives. 5.1 Convergence
Figure 2 showsthat our agents converge and demonstrate consistent training over time, with both performance metrics growing and eventually stabilizing. Note that both the hypervolume and Expected Utility metrics appear large due to their exponential scaling with the number of objectives and the achievable value ranges, particularly influenced by the high values of the GEO objective. 5.2 Solutions
Figure 3 showsthe Pareto set of solutions produced by J US TICE and the single RICE50+ solution (red star). We transform the objective values for simplicity: Total Global Eco
nomic Output representsthe GEO objective, and Global Average Annual Temperature correspondsto the mean of the inverse of IGT (inverted to retrieve the temperature). Global Average Annual Temperature Rise (°C)
Figure 3: Pareto set of policies obtained by JUSTICE with the RICE50+ polict for comparison. Arrows indicate the direction of preference for the objectives. JUSTICE produces 22 solutions (policies), shown as circles in Figure 3. Among these, three are highlighted: Climate Policy (purple), Economic Policy (green), and Compromise Policy (yellow). Although the purple policy is not the most extreme in terms of climate performance, it achieves substantial economic gains with only a slightincrease in temperature compared to the extreme climate solution. Therefore, it is chosen asthe Climate Policy. This example illustrates how the multi-objective approach supports decision-making by presenting a range of trade-offs. The RICE50+ solution lies close to the JUSTICE Pareto front, roughly in the middle, indicating a balance between economy and temperature rise. However, the JUSTICE Compromise Policy dominatesthe RICE50+ solution, albeit by a small margin. Thus, JUSTICE not only offers a range of solutions but also a slightly better solution than RICE50+. 5.3 Comparison of Key Solutions We perform a comprehensive analysis of the highlighted policies in Figure 3—the three JUSTICE policies and the RICE50+ policy. To do so, we plot the key IAM outcomes from 2015 to 2300 for these policies in Figure 4. The economic output trajectories are shown in left panel of Figure 4.
As expected, the best economic policy achievesthe
4.50
4.00
3.50
3.0
Hypervolume Expected Utility
1.70
1.60
1.50
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
Training timesteps Training timesteps
Figure 2: Mean hypervolume and expected utility over training steps (shaded area represents standard deviation). Global Net Economic Output
(Trillion USD PPP)
2000
1500
1000
500
0 2015 2150 2300
Cumulative Abated Emissions
2015 2150 2300
6
5
4
3
2
1
Global Surface
Temperature (°C)
2015 2150 2300
80
60
40
20
0
Compromise Policy Climate Policy Economic Policy RICE50+ Policy
Figure 4: Performance of selected JUSTICE policies over time (years on x-axis) for three importantindicators: Global Net Economic Output, Temperature, and Cumulative Abated Emissions. The indicators show mean values and standard deviation for selected JUSTICE and RICE50+ policies over time. highest output. The best climate policy starts with the lowest output but eventually aligns with RICE50+ levels, as its rapid and deep mitigation efforts stabilize the climate and reduce damages, stimulating economic growth after 2150. The compromise policy presents an interesting trade-off; although it sacrifices slightly more outputin the near term compared to RICE50+, it leadsto significantly higher long-term growth, surpassing RICE50+ by hundreds of trillions of dollars, by reducing climate damagesthrough immediate mitigation. The temperature trajectories are shown in the center panel of Figure 4. As expected, the best economic policy results in the highest temperature rise, while the best climate policy yieldsthe lowest. Under the compromise policy, temperatures exceed 2°C by the end of the 2100s, reaching about 3.5°C by 2300. The RICE50+ temperature projections are significantly lower than those of the JUSTICE compromise policy over the long term. This discrepancy highlights a common criticism of cost-benefit IAMs: their use of simple climate models inadequately capturesthe complexities and uncertainties of climate sensitivity, leading to smooth projectionsthat miss abrupt, high-impact events. Such modelstend to underestimate potential damages and future temperature rises, leading to policy inertia and delayed action. We addressthese issues by using the FAIR model, which captures nonlinear dynamics and feedback mechanisms, effectively handling uncertainty in climate sensitivity. FAIR producestemperature scenariosthat align with more complex Earth System Models featured in the IPCC. The abated emissionstrajectories are shown in the right panel of Figure 4. All three JUSTICE policies favour rapid near-term mitigation, unlike RICE50+, which delays mitigation and peaks only around 2150. This finding aligns with the criticisms of traditional CB-IAMs for undervaluing early mitigation efforts. Among JUSTICE policies, the best climate policy (purple trajectory) achievesthe highest cumulative abated emissions. Both the best economic and compromise policies show similar mitigation levels, though the compromise policy emphasizes earlier action.
This flexibility is crucial when agents have unequal economic capacities and face disproportionate climate impacts in an uncertain future. JUSTICE can have real-world impactin several ways. Decision Support The multi-objective multi-agent nature makes JUSTICE a powerful decision-support tool for climate (e.g., COP) negotiations. In this use case, each JUSTICE agent can represent a country or a coalition with associated preferences, simulating various policy options and implications. This enables stakeholdersto propose, reassess, and refine actions, promoting transparent and robust negotiations. Scientific Discourse Our model can enrich scientific (e.g., IPCC) reports by providing complementary insightsthat capture dimensions often overlooked by conventional models. By exploring additional strategiesthrough multi-objective multi-agent reinforcement learning, our approach broadensthe scope of potential solutions and offers innovative pathways for regionsto understand the consequences of their actions and implement more equitable climate policies. Broader Engagement While IAMs are crucial for policy recommendations, their technical nature and costly software licenses (e.g., models written in GAMS) often limit accessibility. By offering our framework as open source, we empower stakeholdersto explore different scenarios and policy impacts. A more user-friendly interface could enhance this further. Further, our approach bridges research (e.g., IAM and AI) communities, demonstrating the benefits of building on models and algorithms developed by each other. 6.1 Limitations and Directions
The following are key limitations of our work. Addressing these limitations opens interesting avenues for future work. Reward and Utility The MOMARL algorithm we employ makestwo key simplifying assumptions. First, ateam-based reward structure, where all agents receive the same global reward. This is a simplification since, in practice, countries are likely to prioritize their individual interests.
Second, it requires a linear utility function for the multi-objective aspect, which simplifies real-world scenarios. To our knowledge, this isthe only MOMARL algorithm suitable for our setup of continuous state and reward spaces, and a discrete action space). Relaxing these assumptions, i.e., exploring individual reward structures and other utility functions, is an important direction for future work. Scalability We perform analyses considering 12 macroregions of the world. Although this is a substantial improvement over the single-agent assumption of traditional IAMs, a more fine-grained resolution may be necessary to derive insights on equity. This requires a substantial speedup of current MOMARL algorithms. One direction in this regard isto experiment with alternative weight-sampling techniques, such as adaptive weight sampling. Communication Our current model assumes cooperation among regions, whereas real climate negotiations involve complex bargaining and conflicting agendas. Exploring different communication strategies among agents, and modeling trade and capital transfers, could improve the model’s ability to reflect real-world dynamics. Further, training the agents on different climate ensembles of FAIR, which represent varying plausible climate sensitivities, would allow the agentsto take actionsthat are robust under climate uncertainty. Policy Implementation We acknowledge that the policies generated by JUSTICE may be unrealistic, as both RICE50+ and JUSTICE assume mitigation begins at the start of the model run, and the drastic mitigation suggested by JUSTICE is infeasible due to geopolitical dynamics, technological inertia, and policy implementation challenges. However, the utility of JUSTICE is in providing a range of alternatives, which can serve as a scientific basis for stakeholder deliberations. User Studies Finally, assessing which aspects of this framework are most valuable to stakeholders in COP negotiations is crucial. Conducting user studies with decision-makers can offer insights into their perceptions of the results and identify which elements mostinfluence policymaking. 6.2 Conclusions
We introduce the JUSTICE framework, the first to integrate IAM with MOMARL. Our approach stands apart from the modelstypically featured in climate reports—models upon which policymakers rely to inform decisions and negotiate actions. Unlike traditional models, which often oversimplify climate change into problems with a single agent and objective, our model embracesthe complexity of the real world by acknowledging its multi-agent, multi-objective nature. Our experiments show that JUSTICE produces flexible policies and allows detailed exploration of equity compared to RICE50+. Additionally, our model is open-source, implemented in Python, and designed to be accessible for exploration and experimentation of RL algorithms for an important real-world problem.