Modeling Fake News in Social Networks with Deep Multi-Agent Reinforcement Learning

1 INTRODUCTION . Problem setting At least since the 2016 US presidential election , fake news on social networks , aimed at manipulating the users ’ perception of facts , has been recognized as a major issue in open societies . Yet , to date little is understood about how fake news spread on social networks , when a piece of fake news is effective in swaying public opinion and what interventions might be successful in mitigating the effect of fake news . Our lack of understanding of this important phenomenon is in large part due to the difficulty of modeling complex decision making processes on social networks . To tackle this issue , we extend a standard game of information aggregation on a social network , see Mossel et al . ( 2015 ) , to accommodate fake news and solve this , analytically intractable , game using deep multi-agent reinforcement learning . This approach is flexible , computationally tractable , and allows us to work towards answering the questions raised above . Related work Our work relates to two main strands of literature : ( i ) models of information aggregation in social networks and ( ii ) deep multi-agent reinforcement learning as a tool to solve games . The standard information aggregation game , outlined in Section 2.1 , involves agents ( repeatedly ) guessing an unknown state of the world taking into account an informative private signal and observations of actions of their neighbors on the social network in the previous period . Note that this model involves no fake news . There are two main approaches to studying this game . In the rational agent approach , agents ’ decision are best responses and the analysis focuses on the Nash equilibria of the game . In an important contribution , Mossel et al . ( 2015 ) show that under certain conditions on the social network , in the limit T → ∞ ( number of periods ) and n → ∞ ( number of agents ) , agents will converge and will agree on the correct state of the world . This approach is not tractable for finite T and n and therefore does not allow , for example , the study of transient behavior . An alternative approach to studying the game of information aggregation is heuristic , see Golub & Sadler ( 2016 ) for a review of many important contributions . While heuristic approaches are more flexible to changes in the model setup and allow the study of transient behavior , agent strategies can be too simplistic . For example , agents may treat repeated observations of a neighbor ’ s guess as conveying additional information , even when this is not the case ( this occurs for example when this neighbor ’ s only information is his private signal ) . Furthermore , heuristic behavior does not adapt to changing circumstances and would therefore not allow us to study how agent behavior changes in the presence of fake news . Our deep multi-agent reinforcement learning approach combines both worlds . Our approach is flexible since it does not require the model to be analytically tractable and finds near-optimal policies through reinforcement learning . Our work is also related to studies of competitive information diffusion , see Alon et al . ( 2010 ) ; Etesami & Başar ( 2014 ) . However , while the literature on competitive information diffusion focuses on the strategic seeding of a pre-determined diffusion process , our work focuses on modeling the information diffusion process itself through the agents ’ learned actions . Deep multi-agent reinforcement learning ( DMARL ) has been applied to solve games of different structure . Policies or value functions are approximated by deep neural nets and trained via backpropagation . Some recent examples are Letcher et al . ( 2018 ) ; Foerster et al . ( 2018 ) ; Lerer & Peysakhovich ( 2017 ) ; Balduzzi et al . ( 2018 ) . Often the focus is on methods that allow agents to converge on a particular , socially desirable policy , such as cooperation in social dilemmas without changing the payout structure . Since this is not an issue for most of our analyses , we make use of Independent Q-learning , see Tampuu et al . ( 2017 ) . However , for some of our more advanced analyses , convergence does become an issue and applying more advanced DMARL methods provides a promising avenue of future research . Contribution To the best of our knowledge , we are the first to develop a flexible computational model of fake news on social networks in which agents act according to learned best response functions . We achieve this by extending a standard information aggregation model to allow for fake news and by representing agents as recurrent deep Q-networks ( DQN ) . We incorporate fake news into the model by adding an adversarial agent , the attacker , that either provides biased private signals to or takes over a subset of agents . We conduct a number of experiments to answer three main questions . ( i ) What determines the effectiveness of the attacker in reducing the accuracy of information aggregation ? ( ii ) How well can agents learn adapt to the presence of an attacker ? ( iii ) When attacker and agents learn simultaneously , how do policies evolve over time ? Our results suggest three potential interventions to curb the effectiveness of fake news on social networks . First , “ vaccinate ” users of social networks by making them aware of the presence of fake news . Second , keep private information on social networks private so that attackers can not target poorly informed or well connected users . Third , encourage the formation of “ balanced ” rather than clustered or fragmented social networks , so that information can “ flow ” effectively across the network . Lastly , we contribute to the MARL literature by providing an important and concrete real-world application of methods that to date have found limited practical application . 2 BACKGROUND . 2.1 AN INFORMATION AGGREGATION GAME ON A SOCIAL NETWORK . We consider a sequential game of information aggregation over T periods indexed by t = 1 , . . . , T with a set of agents N , see Mossel et al . ( 2015 ) ; Golub & Sadler ( 2016 ) for details . Agents interact via a fixed , possibly directed , graph G ( N , E ) , where E denotes the set of edges connecting the nodes N ; see Figure 1 for an illustration of the networks used in this paper . For concreteness , one can think of the agents as users of a social network , such as Twitter . A directed link from agent A to agent B would then imply that A follows B on Twitter . Agents are presented with a claim that is either true or false . Again , for concreteness , one can think of the claim as a factual statement encountered in a piece of news that has been shared among the agents on the social network.1 True claims are denoted as θ = 1 and false claims as θ = 0 . True and false claims occur with equal probability ; that is , the unconditional probability of a claim being 1 We restrict our analysis to the case when claims are factual statements that can be objectively verified as true or false . Of course this prevents us from studying the dynamics of opinions . However , since we focus on the effect of fake news on the social network , restricting our analysis to factual statements is natural . true or false is P ( θ = 0 ) = P ( θ = 1 ) = 1/2 . This choice simplifies our analysis without affecting any of our qualitative results . Agents receive private signals si ∼ Fθ once at the beginning of an episode . In the baseline Fθ is such that private signals are always informative but noisy . In each period agents choose a binary action ait ∈ { 0 , 1 } . Agents observe the actions of their neighbors on the graph from the previous period . At the end of the episode , agents are rewarded for actions that matched the veracity of the claim presented . For a given claim θ and sequence of actions { ait } , an agent ’ s total discounted reward is ri = ∑T t γ t1 { ait = θ } . Note that in this game , for t < T , it is not necessarily optimal to choose the action that matches the maximum of the posterior likelihood of θ . This is because an agent wants to prevent others from simply copying its action , since that reveals no information about the others ’ private signal . Injecting noise into actions may therefore be beneficial . Despite these concerns , it can be shown that , under certain conditions on the network , in the limit T →∞ ( number of periods ) and n→∞ ( number of agents ) , agents will converge and will agree on the correct θ , see Mossel et al . ( 2015 ) . 2.2 MULTI-AGENT REINFORCEMENT LEARNING : INDEPENDENT Q-LEARNING . In principle , reinforcement learning ( RL ) can be used to learn the agents ’ policy ( or response functions ) in any game . Many methods have been proposed to learn reasonable best response functions in various multi-agent reinforcement learning ( MARL ) settings , see Foerster et al . ( 2016 ) ; Letcher et al . ( 2018 ) ; Lerer & Peysakhovich ( 2017 ) . In this paper , the agents ’ response functions will be learned via independent Q-learning ( IQL ) and Q functions will be approximated by deep recurrent neural networks , see Tampuu et al . ( 2017 ) ; Foerster et al . ( 2016 ) . We relegate a discussion of the RL and IQL framework to Appendix A.1 . 3 METHODS . Graphs We consider three types of undirected graphs , see Figure 1 for an illustration . An instance of a Barabasi-Albert random graph with |N | = 10 , see Barabási & Albert ( 1999 ) ; a clustered graph with three clusters ( nc = 3 ) of fully connected nodes with four nodes each ( sc = 4 ) such that |N | = 12 ; and a balanced graph ( |N | = 12 ) that is obtained by randomly rewiring the clustered graph . Random rewiring involves repeatedly selecting two edges uniformly at random and swapping the terminal nodes of these edges . This preserves the degree sequence , i.e . the number of neighbors of each node , but removes the clusters such that the rewired graph is “ balanced ” . The BarabasiAlbert random graph is often used to model social networks and is therefore a natural choice for our application . The clustered graph is a stereotypical example of a social network which is comprised of cliques that are weakly connected . It thereby represents a more “ fragmented ” or “ polarized ” social network . The balanced graph removes these clusters from the graph while maintaining the degree sequence and can thus help isolate the effect of clusters on information aggregation . Fake news We extend the model of information aggregation in order to study the effects of fake news on information aggregation . We introduce an adversary , the so called attacker , who wants to reduce the other agents ’ total reward ∑ i ri . To disambiguate between the attacker and other agents , we often refer to these other agents as citizens . The attacker wants to persuade citizens to support false claims and dissuade them from supporting true claims . The attacker can achieve this in one of two ways . First , in the biased signal attack , he can distribute a fixed budget of bias β = ∑ i β i across a subset of the agents such that citizen i ’ s private signal is drawn from si ∼ Fθ ( βi ) .2 We choose si ∼ N ( θ + βi ( 1− 2θ ) , σ2 ) . We set σ2 = 1 throughout . In this case the choice of { βi } i∈N is hand-tuned , i.e . set by the experimenter and not optimized . Second , in the agent takeover attack , he can directly take over an agent and act on his behalf . In this case the attacker optimizes his attack policy by IQL . Learning to aggregate information We hold the network fixed across episodes . At the beginning of each episode θ and all private signals are drawn at random according to the distributions given above . For citizens we train a single DQN . In period t , the citizen observes the episode private signal si , his and the actions of his neighbors in the previous period { ajt−1 } j∈Bi ( Bi is the set of neighbors of agent i augmented by i ) and the agent id i which identifies the agent ’ s network position . To maintain a constant length input vector , unobserved actions are encoded as −1 . Thus , the corresponding DQN is a function Q : R× { −1 , 0 , 1 } |N | ×N × { 0 , 1 } 7→ R. In the agent takeover attack , the attacker is represented by a DQN . This DQN is distinct from the citizen DQN . At the beginning of each episode the attacker chooses a citizen uniformly at random from N and acts on his behalf . In period t , the attacker observes θ without noise , his and the actions of his neighbors in the previous period { ajt−1 } j∈Bi and the agent id i . Thus , the corresponding DQN is a function Q : { 0 , 1 } × { −1 , 0 , 1 } |N |×N × { 0 , 1 } 7→ R. In the biased signal attack , the attacker is not represented by a DQN . Instead , the biases are hand-tuned . At the beginning of each episode the attacker chooses a set of citizens uniformly at random from N and delivers biased signals to them as outlined above . Measuring information aggregation Intuitively , agents should be able learn to extract information from their private signals and their neighbors ’ actions , such that at the end of an episode their expected reward ( E [ riT ] ) should be significantly higher than at the beginning of an episode ( E [ ri0 ] ) . That is , agents should aggregate information over time . In the following , we give a formal definition of information aggregation for finite N and T that is useful in our case . Definition 1 ( Information aggregation ) . Let F = { si } i∈N denote the set of all realized signals in a given episode . Let θ̂ = argmaxθP [ θ | F ] denote the maximum-a-posteriori estimator of θ given F . Then information is aggregated if for all i ∈ N , we have aiT = θ̂ . That is , under this definition information is aggregated if agents act as if they had seen all private signals . In reality , it is not reasonable to expect perfect information aggregation with actions learned via independent Q-learning . We therefore define the following measure of information aggregation . Definition 2 ( Accuracy of information aggregation ) . The accuracy of information aggregation at time step t is At = E [ 1 { ait = θ } ] . A useful benchmark to compare At to is of course E [ 1 { θ̂ = θ } ] . Another useful benchmark is E [ 1 { θ̃ = θ } ] , where θ̃ = argmaxθP [ θ | si ] is the MAP estimator given only a single private signal . AT can never exceed E [ 1 { θ̂ = θ } ] and A1 can never exceed E [ 1 { θ̃ = θ } ] and of course E [ 1 { θ̂ = θ } ] > E [ 1 { θ̃ = θ } ] provided |N | > 1 . Once we have trained the agents via Q-learning it is easy to estimate At by evaluating a set of batches of games B and computing Ât = 1/|B|1/|N | ∑ b∈B , i∈N 1 { ait , b = θ } , where ait , b is the action taken by agent i in batch b ∈ B . In the following we will simply refer to Ât as the accuracy . 2 Biased signals can be thought of as targeted ads to users of the social network . Indeed , it been shown that fake news attacks have taken exactly this form , see Chiou & Tucker ( 2018 ) . A fixed bias budget corresponds roughly to an attacker with a fixed budget for targeted social media ads . Heuristic benchmark We compare the accuracy of the DQN against the DeGroot heuristic , DeGroot ( 1974 ) . Under this heuristic information aggregation rule at t = 0 , we set ai0 = θ̃ ( s i ) . In each subsequent period actions are set to at = Uat−1 , where at is the vector of actions taken by citizens and U = D−1 ( I +M ) , where I is the identity matrix , M is the graph adjacency matrix and D is a diagonal matrix with entries Dii being the number of neighbors of node i plus one ( i.e . a self loop ) . Information is therefore aggregated through repeated averaging of past actions .

Summary: This paper proposes a model under which to study social networks under attacks attempting to propagate misinformation. It proposes a theoretical model based on assumptions on what kinds of graphs are common in social media and what kinds of attacks take place. While this could be interesting, the work presented falls short of what it promises, i.e. to develop a practical model of fake news on social networks,  because many of the assumptions made about the phenomena under study are unrealistic. In more detail