Learning Homophilic Incentives in Sequential Social Dilemmas

1 INTRODUCTION . Deep multi-agent reinforcement learning ( MARL ) has achieved prominent progress ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Baker et al. , 2020 ; Wang et al. , 2021 ) . While much effort is devoted to fully cooperative settings , decision-making individuals in many real-world situations may be self-interested , such as autonomous vehicles ( Bonnefon et al. , 2016 ) , taxpayers ( Rothstein , 2001 ) , and governments dealing with climate change ( Capstick , 2013 ) . A unique phenomenon among self-interested agents is social dilemma , in which individually rational behavior results in a situation where everyone suffers in the long run . Historically , the problem of how cooperation emerges in social dilemmas has long perplexed scientists from various fields , such as evolutionary biology , animal behavioristics ( Packer et al. , 1991 ) , and neuroscience ( Rilling et al. , 2002 ) , and is listed as one of the 125 most compelling science questions for the 21st century ( Kennedy , 2005 ; Pennisi , 2005 ) . Cooperation emergence in non-sequential social dilemmas ( SDs ) have been extensively studied ( Boyd , 1989 ; Rand & Nowak , 2011 ) . However , these methods only consider the problems that can be cast as a matrix game ( Hughes et al. , 2018 ) , and typically assume the existence of atomic actions for cooperative and defective strategies . Thus their applicability to real-world , temporally extended SDs is limited , where cooperation and defection are policies that need to be learned . With the ability to learn policies in complex tasks , deep MARL shows promise to fill the gap and study how cooperation emerges in these more realistic sequential social dilemmas ( SSDs ) ( Leibo et al. , 2017 ) . Some previous works have studied SSDs from the perspective of MARL ( Jaques et al. , 2019 ; Hughes et al. , 2018 ) . Among them , Yang et al . ( 2020 ) and Lupu & Precup ( 2020 ) extend ( altruistic ) incentives to temporally extended cases . Referring to individuals paying costs to punish or reward others even though there is no immediate gain by these actions for themselves ( Ostrom et al. , 1992 ; Güth , 1995 ; Fehr & Gächter , 2002 ; Boyd et al. , 2003 ; Mussweiler & Ockenfels , 2013 ) , altruistic incentives have been proven to be highly related to cooperation emergence in non-sequential SDs ( Fowler , 2005 ; Akçay & Roughgarden , 2011 ; Fong & Surti , 2009 ) . However , altruistic incentives have not been fully studied in SSDs . The aforementioned deep methods do not effectively scale up and are limited in very small settings . With further investigation , we find that these methods do not converge to stable cooperation , where the cooperation level regularly oscillates between cooperation and defection . In this paper , we first analyze the underlying reasons behind the phenomenon of such unstable cooperation . We find that , although incentives make cooperation more likely to emerge , they also introduce a second-order social dilemma ( 2nd-SDs ) problem into the system – if someone else would pay costs to punish or reward others , why should I bother to do so ( Dreber et al. , 2008 ) ? The consequence is that more and more second-order free-riders would exploit those agents who make an effort to incentivize others , resulting in a degraded incentivizing mechanism and thus collapsed cooperation . 2nd-SDs have been well studied in non-sequential social dilemmas ( Fowler , 2005 ; Greenwood , 2016 ) . However , solving 2nd-SDs in sequential settings remains largely untouched in the literature and posts new challenges – the identification of cooperation and defection for incentivizing is non-trivial in SSDs and cooperation is more vulnerable to 2nd-SDs in SSDs than in SDs . Thus SSDs require a different computational approach to promote cooperation . Specifically , in SSDs , agents need to learn temporally extended strategies which are much more complex than simple cooperation and defection action in SDs and can be dynamically mixed with cooperation and defection behaviors , especially during the learning . The agents that have successfully learned cooperative incentivizing policies can be exploited by other agents with temporarily learned mixing strategies , and the exploited cooperators may abandon cooperative strategies , making it less likely for other agents to learn to cooperate . This issue does not exist in SDs but will severely hinder cooperation emergence in SSDs . To solve this problem , we first illustrate the dynamics of and formally analyze the impetus for the second-order social dilemma on a fully-featured motivating example . Based on these analyses , we propose a novel learning mechanism by encouraging agents with similar environmental behaviors to have similar incentivizing behaviors . Our method is inspired by a concept called homophily , a particularly common tendency for human individuals to associate or bound with similar others , as in the proverb birds of a feather flock together . In the literature , homophily has been studied in non-sequential social dilemmas , where agents interact with others of similar predefined types ( Fu et al. , 2012 ; Fletcher & Doebeli , 2009 ; Ramazi et al. , 2016 ) . In non-sequential cases , agents can choose their partners to interact with ( Rand et al. , 2011 ; Aksoy , 2015 ) or there is a matching system that auto-matches similar agents ( Bergstrom , 2003 ; Bilancini et al. , 2016 ) , but it is not clear how to extend these insights to SSDs . Our work solves this problem and enables a group of self-interested agents to cooperate stably when they are continually learning and acting in a shared environment . It is worth noting that multi-person social dilemmas are classified into two broad categories : public goods and the tragedy of the commons ( HARDİN , 1968 ; Kollock , 1998 ; Ledyard , 1994 ; Hardin , 2009 ) . We evaluate our method on both of these sequential social dilemmas and show that the agent population learns stable cooperative behaviors with great efficiency and stability . Visualization of the evolution process of cooperation shows that homophily effectively enables stable cooperation by preventing agents that conduct altruistic incentives from being exploited by second-order free-riders . 2 PRELIMINARIES AND RELATED WORKS . Social dilemmas ( SDs ) are among the most important settings used to study the emergence of cooperation . In a social dilemma , there exists a tension between the individual and collective rationality , in which individually rational behavior results in a situation where everyone suffers in the long run . Although studies on SDs have contributed significantly to the research of cooperation emergence for decades ( Axelrod & Hamilton , 1987 ; Peysakhovich & Lerer , 2017 ; Anastassacos et al. , 2020 ) , they focus on fixed policies , where the cooperative and defective strategies are actions to be chosen rather than policies to be learned . To be more realistic as in real-world situations , in this paper , we consider sequential social dilemmas ( SSDs , ( Leibo et al. , 2017 ; Blanco et al. , 2014 ) ) . An SSD can be modelled as a partially-observable general-sum Markov game ( Littman , 1994 ) M = 〈I , S , { Ai } , P , O , { Ωi } , { Ri } , n , γ〉 , where I is a finite set of n agents and γ is a discount factor . At each time step , agent i draws a partial observation oi ∈ Ωi of the state s ∈ S according to the observation function O ( s , i ) . Based on the observation , agent i selects an action ai ∈ Ai , which together form a joint action a , leading to a next state s′ according to the stochastic transition function P ( s′|s , a ) and individual rewards ri = Ri ( s , a ) for each agent . In SSDs , instead of taking atomic cooperation or defection actions , agents must learn cooperation or defection strategies consisting of potentially long sequences of environmental actions . The goal of each agent is to maximize the local expected return : Qi ( s , a ) = Es0 : ∞ , a0 : ∞ [ ∑∞ t=0 γ tRi ( st , at ) |s0 = s , a0 = a ] . Altruistic incentive , including altruistically rewarding and punishing others , is known as one of the solutions for social dilemmas ( Kollock , 1998 ) . However , it introduces the problem of second-order social dilemma ( 2nd-SD ) , which arises from each individual ’ s inclination to free ride on a mechanism that is designed to solve the first-order social dilemma . To solve 2nd-SDs , previous works either introduce additional mechanisms , such as extra punishing mechanisms ( Fowler , 2005 ; Greenwood , 2016 ) , reputation mechanisms ( Panchanathan & Boyd , 2004 ) , or change the game settings , such as enabling the amount of public goods to grow exponentially with the number of contributors ( Ye et al. , 2016 ) , considering corruption and power asymmetries ( Úbeda & Duéñez-Guzmán , 2011 ) . These works analyze the dynamics of predefined and fixed mechanisms and focus on non-sequential games . We propose to solve 2nd-SDs in general cases using the tendency of homophily , which states that agents tend to have similar incentivizing behaviors if their environmental behaviors are similar . In the literature , homophily , frequently termed as assortative matching ( Bergstrom , 2003 ) , is often referred to the tendency for agents to interact with others of similar types ( Fu et al. , 2012 ; Fletcher & Doebeli , 2009 ; Ramazi et al. , 2016 ) . These works assume that a game ( such as Prisoner ’ s Dilemma and Public Goods Dilemma ) involves only a group of several agents from a large population . Based on the way they form the group , these works can be divided into two broad categories . ( 1 ) Agents can choose their partners to interact with ( Rand et al. , 2011 ; Aksoy , 2015 ) . However , in SSDs , agents do not have such an option and their partners are determined at initialization . They have to continually learn and act in a shared environment instead of a separate environment instance for each group . ( 2 ) There is a matching system that can auto-match agents with similar predefined types to join the game ( Bergstrom , 2003 ; Bilancini et al. , 2016 ) . However , in SSDs , agents are assumed to have determined partners and thus the matching process changes the game settings . Moreover , the existence of such a matching system is an unrealistic assumption because the policy types ( such as cooperation and defection ) in SSDs can not be predefined since temporally extended strategies are much more complex and can be dynamically mixed with cooperation and defection , especially during learning . Therefore , previous work studying homophily can not be applied to SSDs . Moreover , Wang et al . ( 2018 ) also find that assortative matching of environmental behaviors can not promote cooperation in SSDs . In contrast , our work encourages homophily on the level of incentivizing behaviors , which we find outperforming other algorithms on learning cooperation in SSDs . Furthermore , we discuss how homophily relates to other possible solutions in Appendix B.9 . Recent works study some other mechanisms which may encourage cooperation in SSDs . Hughes et al . ( 2018 ) study the effects of inequality aversion , which bypasses the problem of second-order social dilemmas because the punishment does not incur costs to any agents other than the punished ones . Jaques et al . ( 2019 ) find that encouraging mutual influence among agents can promote collective behaviors . We empirically compare with these methods in Sec . 5.2 . In the following sections , we will first provide a motivating example to show the influence of SDs , 2nd-SDs , and how homophily can solve 2nd-SDs . Based on these analyses , we introduce how to encourage homophily in temporally extended cases in Sec . 4 .

Summary: This paper studies the problem of promoting cooperation among self-interested agents. Inspired by the work which studied homophily in non-sequential social dilemmas, the authors proposed to use homophily, which encourages agents with similar environment behaviors to have similar incentivizing behaviors, to achieve stable cooperation in sequential social dilemmas. Simulation evidence from sequential social dilemmas of public goods and tragedy of the commons was also presented in the paper to support the effectiveness of the proposed method.