Title: LOPT: Learning Optimal Pigovian Tax in Sequential Social Dilemmas
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-agent Reinforcement Learning

Sequential Social Dilemma
Abstract: Multi-agent reinforcement learning (MARL) has emerged as a powerful framework for modeling autonomous agents that independently optimize their individual objectives. However, in mixed-motive MARL environments, rational self-interested behaviors often lead to collectively suboptimal outcomes situations commonly referred to as social dilemmas.
A key challenge in addressing social dilemmas lies in accurately quantifying and representing them in a numerical form that captures how self-interested agent behaviors impact social welfare.
To address this challenge, \textit{externalities} in the economic concept is adopted and extended to denote the unaccounted-for impact of one agent's actions on others, as a means to rigorously quantify social dilemmas.
Based on this measurement, a novel method, \textbf{L}earning \textbf{O}ptimal \textbf{P}igovian \textbf{T}ax (\textbf{LOPT}) is proposed. Inspired by Pigovian taxes, which are designed to internalize externalities by imposing cost on negative societal impacts, LOPT employs an auxiliary tax agent that learns an optimal Pigovian tax policy to reshape individual rewards aligned with social welfare, thereby promoting agent coordination and mitigating social dilemmas. We support LOPT with theoretical analysis and validate it on standard MARL benchmarks, including Escape Room and Cleanup. Results show that by effectively internalizing externalities that quantify social dilemmas, LOPT aligns individual objectives with collective goals, significantly improving social welfare over state-of-the-art baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Learning Optimal Pigovian Tax (LOPT), a framework to quantitatively address social dilemmas in mixed-motive multi-agent reinforcement learning (MARL). LOPT is inspired from economics, where externalities are effectively modeled into social dilemmas. Specifically, LOPT uses an agent the authors term a tax planner, which learns to allocate allowances to reflect values of externalities. The LOPT framework is validated through theoretical analyses and on MARL benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n(+) The paper is generally well-written and easy to follow. \n\n(+) The notion of introducing externalities from economics to quantify social dilemmas in MARL is novel. \n\n(+) Theoretical analysis to demonstrate that LOPT can effectively quantity externalities. \n\n(+) Intuitive experiment domains to demonstrate the validity of the proposed approach. \n\n(+) Evaluation on multiple baselines showcase improvement of LOPT over other methods. \n\nAreas for improvement: \n\n(-) It is not clearly mentioned how the LOPT formulation in Sec. 2-3 is different from other widely examined reward shaping approaches, including techniques such as potential-based reward shaping. \n\n(-) A critical result when using methods like potential based reward shaping is that the identity of optimal policies are preserved- i.e., the optimal policy learned when using reward shaping is the same as the optimal policy in the original environment without reward shaping. Does LOPT provide an analogous guarantee? \n\n(-) While the experimental evaluations are on intuitive environments, the large body of work examining reward shaping and credit assignment in MARL showcase verification on complex multi-agent domains such as StarCraft. Is there a scope for evaluating LOPT in these domains in order to provide a more thorough comparison? \n\n(-) The readability of Fig. 5 and Fig. 6 can be significantly improved. In the current version, it is impossible to see the legends without zooming in and further, the shaded regions showing variance are difficult to discern.\n\nQUESTIONS:\nQ1: It is not clearly mentioned how the LOPT formulation in Sec. 2-3 is different from other widely examined reward shaping approaches, including techniques such as potential-based reward shaping. Could the authors comment? \n\nQ2: A critical result when using methods like potential based reward shaping is that the identity of optimal policies are preserved- i.e., the optimal policy learned when using reward shaping is the same as the optimal policy in the original environment without reward shaping. Does LOPT provide an analogous guarantee? \n\nQ3: While the experimental evaluations are on intuitive environments, the large body of work examining reward shaping and credit assignment in MARL showcase verification on complex multi-agent domains such as StarCraft. Is there a scope for evaluating LOPT in these domains in order to provide a more thorough comparison?\n\nLIMITATIONS:\nAddressed in the Conclusion very briefly. The authors could expand on this, and use a separate Limitations section/ subsection.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new model for MARL in which the reward of each agent is shaped to consider social welfare. The new term is inspired by Pigovian tax and is learned using an external agent using gradient descent towards the optimal tax. The paper includes two theorems that together form a theoretical basis for the learned tax to be optimal. Empirical evaluation includes two domains and a variety of agents that are self-interested. The results show that the new method, LOPT, enables the agents to learn a balanced policy that maintains social welfare.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ Looking at shaping social welfare into MARL via a Pigovian tax seems novel\n+ The background and the explanations for the new method are clear and informative\n+ Empirical evaluation seems sound, and results are consistent with the claims\n\nWeaknesses:\n- Notation is inconsistent and sometimes unexplained. For example, in line 159 it seems that the last indexing of r should be (-j) rather than j. Similarly, the notation in line 201 with pi^p_{\\Phi_p} but then the same term in Eq. 9 is pi_p^{\\Phi_p}. Also, I'm unsure that the numbers in Figure 2 depicting the prisoner's dilemma are correct.\n- Theorems are considered a part of the main paper, but the proofs are solely in the appendix, so they cannot be considered a fully sound contribution.\n- Related work section is missing on some substantial recent work in the field of solving social dilemmas using MARL, including for example:\n1. Li, Yang, et al. \"Aligning Individual and Collective Objectives in Multi-Agent Cooperation.\" The Thirty-eighth Annual Conference on Neural Information Processing Systems.\n2. Haupt, Andreas, et al. \"Formal contracts mitigate social dilemmas in multi-agent reinforcement learning.\" Autonomous Agents and Multi-Agent Systems 38.2 (2024): 1-38.\n- The explanation of the experimental design is missing and hard to follow if you're not familiar with the domain. For example, can an agent pull the level from every place, or just if they opened the door? Given the current explanation, why would anyone pull the lever?\n\nQUESTIONS:\n1. In the prisoner's dilemma, shouldn't all values be negative except the 0s in the original formulation, and except the 8s in the corners in the matrix after the tax?\n2. What happens if there is some correlation between x_i and x_j? For example, having everyone use the same power grid simultaneously will cause it to collapse, even though the same usage would not incur such a penalty if it were spread out over time.\n3. Given the current explanation for the escape room domain, why would anyone pull the lever?\n\nLIMITATIONS:\nIt can be beneficial to have a short discussion about dual purpose in this research: can someone use the same paradigm as LOPT to *decrease* social welfare to intentionally cause unrest?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a MARL paradigm to escape the social dilemma in multi-agent sequential games. The method is inspired by Pigovian taxes theory and build the structure of a high-level tax planner into the MARL algorithm. The paper first established the exisitng theory and built its connection with the MARL setting. Then, with the proposed approach, it conducts experiments on classic social dilemma games. The results show that the MARL can successfully escape such local minimal and explains the implication of the tax framework to the games.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. The idea of building the Pigovian taxes bias into the MARL algorithm is interesting and the motivation is easy to understand.\n3. The proof of concept experiment is comprehensive in analysis and is supportive to the pipeline.\n\na. The empirical results suggest that the proposed frame work is very strong, e.g, the Escape Room, where the policy is able to converge in just a few steps while the existing baselines struggles.\n\nb. The method has been shown to be able to generalize to more complicated setting including 5 agents and 18x25 size, which suggests certain scalability.\n\nWeakness:\n\n1. Apart from the existing proof of concept experiments, it is unknown of the scalability of the MARL algorithm. For example, with significantly more agents or the control space is larger or no longer discrete. I would imagine that doing similar experiments with significantly more agents and grid size would make the work much more impactful.\n2. The paper uses very small figure which make the reader hard to read. i would suggest reducing the narrative and use larger figures and more illustrations including the visualization of the policy of the proposed approach.\n\nQUESTIONS:\n1. Is it possible to extend the algorithm to continuous control space?\n2. How is the scalability of the proposed method of like N=20 agents? Does it significantly take more compute?\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
A key challenge in multi-agent settings is how to address social dilemmas - the scenario whereby rational agents acting in their own self interest produce suboptimal social outcomes. The authors aim to address this issue by training an AI agent to apply reward shaping to internalise the social costs produced by each agent, thereby applying a 'pigovian tax'. Under their problem formulation, the authors show the existence of an optimal solution and then demonstrate empirically the performance of a policy gradient method with their propose reward shaping formulation.

STRENGTHS AND WEAKNESSES:
Strengths:
- The formulation and proofs of existence seem correct as far as I can tell.
- The experiments compare to a good range of baselines.
- 

Weaknesses:
- The paper could be more clearly written

QUESTIONS:
- I may have missed this, but how many runs are each of the results averaged over?
- How many samples are required to develop such a tax planning agent? Is this reasonable in real environments?

LIMITATIONS:
There is limited discussion on the limitations of their work
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03