Learning Homophilic Incentives in Sequential Social Dilemmas

1 INTRODUCTION . Deep multi-agent reinforcement learning ( MARL ) has achieved prominent progress ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Baker et al. , 2020 ; Wang et al. , 2021 ) . While much effort is devoted to fully cooperative settings , decision-making individuals in many real-world situations may be self-interested , such as autonomous vehicles ( Bonnefon et al. , 2016 ) , taxpayers ( Rothstein , 2001 ) , and governments dealing with climate change ( Capstick , 2013 ) . A unique phenomenon among self-interested agents is social dilemma , in which individually rational behavior results in a situation where everyone suffers in the long run . Historically , the problem of how cooperation emerges in social dilemmas has long perplexed scientists from various fields , such as evolutionary biology , animal behavioristics ( Packer et al. , 1991 ) , and neuroscience ( Rilling et al. , 2002 ) , and is listed as one of the 125 most compelling science questions for the 21st century ( Kennedy , 2005 ; Pennisi , 2005 ) . Cooperation emergence in non-sequential social dilemmas ( SDs ) have been extensively studied ( Boyd , 1989 ; Rand & Nowak , 2011 ) . However , these methods only consider the problems that can be cast as a matrix game ( Hughes et al. , 2018 ) , and typically assume the existence of atomic actions for cooperative and defective strategies . Thus their applicability to real-world , temporally extended SDs is limited , where cooperation and defection are policies that need to be learned . With the ability to learn policies in complex tasks , deep MARL shows promise to fill the gap and study how cooperation emerges in these more realistic sequential social dilemmas ( SSDs ) ( Leibo et al. , 2017 ) . Some previous works have studied SSDs from the perspective of MARL ( Jaques et al. , 2019 ; Hughes et al. , 2018 ) . Among them , Yang et al . ( 2020 ) and Lupu & Precup ( 2020 ) extend ( altruistic ) incentives to temporally extended cases . Referring to individuals paying costs to punish or reward others even though there is no immediate gain by these actions for themselves ( Ostrom et al. , 1992 ; Güth , 1995 ; Fehr & Gächter , 2002 ; Boyd et al. , 2003 ; Mussweiler & Ockenfels , 2013 ) , altruistic incentives have been proven to be highly related to cooperation emergence in non-sequential SDs ( Fowler , 2005 ; Akçay & Roughgarden , 2011 ; Fong & Surti , 2009 ) . However , altruistic incentives have not been fully studied in SSDs . The aforementioned deep methods do not effectively scale up and are limited in very small settings . With further investigation , we find that these methods do not converge to stable cooperation , where the cooperation level regularly oscillates between cooperation and defection . In this paper , we first analyze the underlying reasons behind the phenomenon of such unstable cooperation . We find that , although incentives make cooperation more likely to emerge , they also introduce a second-order social dilemma ( 2nd-SDs ) problem into the system – if someone else would pay costs to punish or reward others , why should I bother to do so ( Dreber et al. , 2008 ) ? The consequence is that more and more second-order free-riders would exploit those agents who make an effort to incentivize others , resulting in a degraded incentivizing mechanism and thus collapsed cooperation . 2nd-SDs have been well studied in non-sequential social dilemmas ( Fowler , 2005 ; Greenwood , 2016 ) . However , solving 2nd-SDs in sequential settings remains largely untouched in the literature and posts new challenges – the identification of cooperation and defection for incentivizing is non-trivial in SSDs and cooperation is more vulnerable to 2nd-SDs in SSDs than in SDs . Thus SSDs require a different computational approach to promote cooperation . Specifically , in SSDs , agents need to learn temporally extended strategies which are much more complex than simple cooperation and defection action in SDs and can be dynamically mixed with cooperation and defection behaviors , especially during the learning . The agents that have successfully learned cooperative incentivizing policies can be exploited by other agents with temporarily learned mixing strategies , and the exploited cooperators may abandon cooperative strategies , making it less likely for other agents to learn to cooperate . This issue does not exist in SDs but will severely hinder cooperation emergence in SSDs . To solve this problem , we first illustrate the dynamics of and formally analyze the impetus for the second-order social dilemma on a fully-featured motivating example . Based on these analyses , we propose a novel learning mechanism by encouraging agents with similar environmental behaviors to have similar incentivizing behaviors . Our method is inspired by a concept called homophily , a particularly common tendency for human individuals to associate or bound with similar others , as in the proverb birds of a feather flock together . In the literature , homophily has been studied in non-sequential social dilemmas , where agents interact with others of similar predefined types ( Fu et al. , 2012 ; Fletcher & Doebeli , 2009 ; Ramazi et al. , 2016 ) . In non-sequential cases , agents can choose their partners to interact with ( Rand et al. , 2011 ; Aksoy , 2015 ) or there is a matching system that auto-matches similar agents ( Bergstrom , 2003 ; Bilancini et al. , 2016 ) , but it is not clear how to extend these insights to SSDs . Our work solves this problem and enables a group of self-interested agents to cooperate stably when they are continually learning and acting in a shared environment . It is worth noting that multi-person social dilemmas are classified into two broad categories : public goods and the tragedy of the commons ( HARDİN , 1968 ; Kollock , 1998 ; Ledyard , 1994 ; Hardin , 2009 ) . We evaluate our method on both of these sequential social dilemmas and show that the agent population learns stable cooperative behaviors with great efficiency and stability . Visualization of the evolution process of cooperation shows that homophily effectively enables stable cooperation by preventing agents that conduct altruistic incentives from being exploited by second-order free-riders . 2 PRELIMINARIES AND RELATED WORKS . Social dilemmas ( SDs ) are among the most important settings used to study the emergence of cooperation . In a social dilemma , there exists a tension between the individual and collective rationality , in which individually rational behavior results in a situation where everyone suffers in the long run . Although studies on SDs have contributed significantly to the research of cooperation emergence for decades ( Axelrod & Hamilton , 1987 ; Peysakhovich & Lerer , 2017 ; Anastassacos et al. , 2020 ) , they focus on fixed policies , where the cooperative and defective strategies are actions to be chosen rather than policies to be learned . To be more realistic as in real-world situations , in this paper , we consider sequential social dilemmas ( SSDs , ( Leibo et al. , 2017 ; Blanco et al. , 2014 ) ) . An SSD can be modelled as a partially-observable general-sum Markov game ( Littman , 1994 ) M = 〈I , S , { Ai } , P , O , { Ωi } , { Ri } , n , γ〉 , where I is a finite set of n agents and γ is a discount factor . At each time step , agent i draws a partial observation oi ∈ Ωi of the state s ∈ S according to the observation function O ( s , i ) . Based on the observation , agent i selects an action ai ∈ Ai , which together form a joint action a , leading to a next state s′ according to the stochastic transition function P ( s′|s , a ) and individual rewards ri = Ri ( s , a ) for each agent . In SSDs , instead of taking atomic cooperation or defection actions , agents must learn cooperation or defection strategies consisting of potentially long sequences of environmental actions . The goal of each agent is to maximize the local expected return : Qi ( s , a ) = Es0 : ∞ , a0 : ∞ [ ∑∞ t=0 γ tRi ( st , at ) |s0 = s , a0 = a ] . Altruistic incentive , including altruistically rewarding and punishing others , is known as one of the solutions for social dilemmas ( Kollock , 1998 ) . However , it introduces the problem of second-order social dilemma ( 2nd-SD ) , which arises from each individual ’ s inclination to free ride on a mechanism that is designed to solve the first-order social dilemma . To solve 2nd-SDs , previous works either introduce additional mechanisms , such as extra punishing mechanisms ( Fowler , 2005 ; Greenwood , 2016 ) , reputation mechanisms ( Panchanathan & Boyd , 2004 ) , or change the game settings , such as enabling the amount of public goods to grow exponentially with the number of contributors ( Ye et al. , 2016 ) , considering corruption and power asymmetries ( Úbeda & Duéñez-Guzmán , 2011 ) . These works analyze the dynamics of predefined and fixed mechanisms and focus on non-sequential games . We propose to solve 2nd-SDs in general cases using the tendency of homophily , which states that agents tend to have similar incentivizing behaviors if their environmental behaviors are similar . In the literature , homophily , frequently termed as assortative matching ( Bergstrom , 2003 ) , is often referred to the tendency for agents to interact with others of similar types ( Fu et al. , 2012 ; Fletcher & Doebeli , 2009 ; Ramazi et al. , 2016 ) . These works assume that a game ( such as Prisoner ’ s Dilemma and Public Goods Dilemma ) involves only a group of several agents from a large population . Based on the way they form the group , these works can be divided into two broad categories . ( 1 ) Agents can choose their partners to interact with ( Rand et al. , 2011 ; Aksoy , 2015 ) . However , in SSDs , agents do not have such an option and their partners are determined at initialization . They have to continually learn and act in a shared environment instead of a separate environment instance for each group . ( 2 ) There is a matching system that can auto-match agents with similar predefined types to join the game ( Bergstrom , 2003 ; Bilancini et al. , 2016 ) . However , in SSDs , agents are assumed to have determined partners and thus the matching process changes the game settings . Moreover , the existence of such a matching system is an unrealistic assumption because the policy types ( such as cooperation and defection ) in SSDs can not be predefined since temporally extended strategies are much more complex and can be dynamically mixed with cooperation and defection , especially during learning . Therefore , previous work studying homophily can not be applied to SSDs . Moreover , Wang et al . ( 2018 ) also find that assortative matching of environmental behaviors can not promote cooperation in SSDs . In contrast , our work encourages homophily on the level of incentivizing behaviors , which we find outperforming other algorithms on learning cooperation in SSDs . Furthermore , we discuss how homophily relates to other possible solutions in Appendix B.9 . Recent works study some other mechanisms which may encourage cooperation in SSDs . Hughes et al . ( 2018 ) study the effects of inequality aversion , which bypasses the problem of second-order social dilemmas because the punishment does not incur costs to any agents other than the punished ones . Jaques et al . ( 2019 ) find that encouraging mutual influence among agents can promote collective behaviors . We empirically compare with these methods in Sec . 5.2 . In the following sections , we will first provide a motivating example to show the influence of SDs , 2nd-SDs , and how homophily can solve 2nd-SDs . Based on these analyses , we introduce how to encourage homophily in temporally extended cases in Sec . 4 .

Summary: How cooperative behavior emergeing among self-interested agents was a question of longstanding interest. In recent years, this problem has been investigated in multi-agent RL situations. This paper considers the instability of the cooperation in MARL.  The paper first illustrates the phenomenon of oscillation-producing second-order social dilemmas within an EGT case study, and shows how a homophily mechanism can resolve them and produce convergence to a socially efficient set of roles. Then they propose a loss function called “homophily loss” that can resolve these second-order dilemmas in MARL. They experiment with this in two SSD's, Harvest and Cleanup, and show that their method outperforms the baselines they investigate.