Title: MODELING FAKE NEWS IN SOCIAL NETWORKS WITH DEEP MULTI-AGENT REINFORCEMENT LEARNING Anonymous authors Paper under double-blind review

Abstract: Over the last few years deep multi-agent reinforcement learning (DMARL) has become an increasingly active area of research with hundreds of papers submitted to top machine learning conferences every year. However, so far there have been few real world use cases that benefited from the progress in the field. We use DMARL to develop a flexible computational model of fake news on social networks in which agents act according to learned best response functions. We achieve this by extending an information aggregation game to allow for fake news and by representing agents as recurrent deep Q-networks (DQN). In the game, agents repeatedly guess whether a claim is true or false taking into account an informative private signal and observations of actions of their neighbors on the social network in the previous period. We incorporate fake news into the model by adding an adversarial agent, the attacker, that either provides biased private signals to, or takes over, a subset of agents. The attacker can follow either a hand-tuned or trained policy. Our model allows us to tackle questions that are analytically intractable in fully rational models, while ensuring that agents follow reasonable best response functions. Our results highlight the importance of awareness, privacy and social connectivity in curbing the adverse effects of fake news and open an entire new real world application area for DMARL.

1. INTRODUCTION:

Problem setting At least since the 2016 US presidential election, fake news on social networks, aimed at manipulating the users' perception of facts, has been recognized as a major issue in open societies. Yet, to date little is understood about how fake news spread on social networks, when a piece of fake news is effective in swaying public opinion and what interventions might be successful in mitigating the effect of fake news. Our lack of understanding of this important phenomenon is in large part due to the difficulty of modeling complex decision making processes on social networks. To tackle this issue, we extend a standard game of information aggregation on a social network, see Mossel et al. (2015) , to accommodate fake news and solve this, analytically intractable, game using deep multi-agent reinforcement learning. This approach is flexible, computationally tractable, and allows us to work towards answering the questions raised above.

Related work Our work relates to two main strands of literature: (i) models of information aggregation in social networks and (ii) deep multi-agent reinforcement learning as a tool to solve games. The standard information aggregation game, outlined in Section 2.1, involves agents (repeatedly) guessing an unknown state of the world taking into account an informative private signal and observations of actions of their neighbors on the social network in the previous period. Note that this model involves no fake news. There are two main approaches to studying this game. In the rational agent approach, agents' decision are best responses and the analysis focuses on the Nash equilibria of the game. In an important contribution, Mossel et al. (2015) show that under certain conditions on the social network, in the limit T → ∞ (number of periods) and n → ∞ (number of agents), agents will converge and will agree on the correct state of the world. This approach is not tractable for finite T and n and therefore does not allow, for example, the study of transient behavior. An alternative approach to studying the game of information aggregation is heuristic, see Golub & Sadler (2016) for a review of many important contributions. While heuristic approaches are more flexible to changes in the model setup and allow the study of transient behavior, agent strategies can be too simplistic. For example, agents may treat repeated observations of a neighbor's guess as conveying additional information, even when this is not the case (this occurs for example when this neighbor's only information is his private signal). Furthermore, heuristic behavior does not adapt to changing circumstances and would therefore not allow us to study how agent behavior changes in the presence of fake news. Our deep multi-agent reinforcement learning approach combines both worlds. Our approach is flexible since it does not require the model to be analytically tractable and finds near-optimal policies through reinforcement learning.

Our work is also related to studies of competitive information diffusion, see Alon et al. (2010); Etesami & Bas ¸ar (2014) . However, while the literature on competitive information diffusion focuses on the strategic seeding of a pre-determined diffusion process, our work focuses on modeling the information diffusion process itself through the agents' learned actions.

Deep multi-agent reinforcement learning (DMARL) has been applied to solve games of different structure. Policies or value functions are approximated by deep neural nets and trained via backpropagation. Some recent examples are Letcher et al. (2018) ; Foerster et al. (2018) ; Lerer & Peysakhovich (2017) ; Balduzzi et al. (2018) . Often the focus is on methods that allow agents to converge on a particular, socially desirable policy, such as cooperation in social dilemmas without changing the payout structure. Since this is not an issue for most of our analyses, we make use of Independent Q-learning, see Tampuu et al. (2017) . However, for some of our more advanced analyses, convergence does become an issue and applying more advanced DMARL methods provides a promising avenue of future research.

Contribution To the best of our knowledge, we are the first to develop a flexible computational model of fake news on social networks in which agents act according to learned best response functions. We achieve this by extending a standard information aggregation model to allow for fake news and by representing agents as recurrent deep Q-networks (DQN). We incorporate fake news into the model by adding an adversarial agent, the attacker, that either provides biased private signals to or takes over a subset of agents. We conduct a number of experiments to answer three main questions. (i) What determines the effectiveness of the attacker in reducing the accuracy of information aggregation? (ii) How well can agents learn adapt to the presence of an attacker? (iii) When attacker and agents learn simultaneously, how do policies evolve over time? Our results suggest three potential interventions to curb the effectiveness of fake news on social networks. First, "vaccinate" users of social networks by making them aware of the presence of fake news. Second, keep private information on social networks private so that attackers cannot target poorly informed or well connected users. Third, encourage the formation of "balanced" rather than clustered or fragmented social networks, so that information can "flow" effectively across the network.

Lastly, we contribute to the MARL literature by providing an important and concrete real-world application of methods that to date have found limited practical application.

2. BACKGROUND:



2.1. AN INFORMATION AGGREGATION GAME ON A SOCIAL NETWORK:

We consider a sequential game of information aggregation over T periods indexed by t = 1, . . . , T with a set of agents N , see Mossel et al. (2015) ; Golub & Sadler (2016) for details. Agents interact via a fixed, possibly directed, graph G(N, E), where E denotes the set of edges connecting the nodes N ; see Figure 1 for an illustration of the networks used in this paper. For concreteness, one can think of the agents as users of a social network, such as Twitter. A directed link from agent A to agent B would then imply that A follows B on Twitter.

Agents are presented with a claim that is either true or false. Again, for concreteness, one can think of the claim as a factual statement encountered in a piece of news that has been shared among the agents on the social network.1 True claims are denoted as θ = 1 and false claims as θ = 0. True and false claims occur with equal probability; that is, the unconditional probability of a claim being true or false is P(θ = 0) = P(θ = 1) = 1/2. This choice simplifies our analysis without affecting any of our qualitative results. Agents receive private signals s i ∼ F θ once at the beginning of an episode. In the baseline F θ is such that private signals are always informative but noisy. In each period agents choose a binary action a i t ∈ {0, 1}. Agents observe the actions of their neighbors on the graph from the previous period. At the end of the episode, agents are rewarded for actions that matched the veracity of the claim presented. For a given claim θ and sequence of actions {a i t }, an agent's total discounted reward is r i = T t γ t 1{a i t = θ}. Note that in this game, for t < T , it is not necessarily optimal to choose the action that matches the maximum of the posterior likelihood of θ. This is because an agent wants to prevent others from simply copying its action, since that reveals no information about the others' private signal. Injecting noise into actions may therefore be beneficial. Despite these concerns, it can be shown that, under certain conditions on the network, in the limit T → ∞ (number of periods) and n → ∞ (number of agents), agents will converge and will agree on the correct θ, see Mossel et al. (2015) .

2.2. MULTI-AGENT REINFORCEMENT LEARNING: INDEPENDENT Q-LEARNING:

In principle, reinforcement learning (RL) can be used to learn the agents' policy (or response functions) in any game. Many methods have been proposed to learn reasonable best response functions in various multi-agent reinforcement learning (MARL) settings, see Foerster et al. (2016) ; Letcher et al. (2018) ; Lerer & Peysakhovich (2017) . In this paper, the agents' response functions will be learned via independent Q-learning (IQL) and Q functions will be approximated by deep recurrent neural networks, see Tampuu et al. (2017) ; Foerster et al. (2016) . We relegate a discussion of the RL and IQL framework to Appendix A.1.

3. METHODS:

Graphs We consider three types of undirected graphs, see Figure 1 for an illustration. An instance of a Barabasi-Albert random graph with |N | = 10, see Barabási & Albert (1999) ; a clustered graph with three clusters (n c = 3) of fully connected nodes with four nodes each (s c = 4) such that |N | = 12; and a balanced graph (|N | = 12) that is obtained by randomly rewiring the clustered graph. Random rewiring involves repeatedly selecting two edges uniformly at random and swapping the terminal nodes of these edges. This preserves the degree sequence, i.e. the number of neighbors of each node, but removes the clusters such that the rewired graph is "balanced". The Barabasi-Albert random graph is often used to model social networks and is therefore a natural choice for our application. The clustered graph is a stereotypical example of a social network which is comprised of cliques that are weakly connected. It thereby represents a more "fragmented" or "polarized" social network. The balanced graph removes these clusters from the graph while maintaining the degree sequence and can thus help isolate the effect of clusters on information aggregation.

Fake news:

We extend the model of information aggregation in order to study the effects of fake news on information aggregation. We introduce an adversary, the so called attacker, who wants to reduce the other agents' total reward i r i . To disambiguate between the attacker and other agents, we often refer to these other agents as citizens. The attacker wants to persuade citizens to support false claims and dissuade them from supporting true claims. The attacker can achieve this in one of two ways. First, in the biased signal attack, he can distribute a fixed budget of bias β = i β i across a subset of the agents such that citizen i's private signal is drawn from

s i ∼ F θ (β i ). 2 We choose s i ∼ N (θ + β i (1 -2θ), σ 2 ).

We set σ2 = 1 throughout. In this case the choice of {β i } i∈N is hand-tuned, i.e. set by the experimenter and not optimized. Second, in the agent takeover attack, he can directly take over an agent and act on his behalf. In this case the attacker optimizes his attack policy by IQL.

Learning to aggregate information We hold the network fixed across episodes. At the beginning of each episode θ and all private signals are drawn at random according to the distributions given above. For citizens we train a single DQN. In period t, the citizen observes the episode private signal s i , his and the actions of his neighbors in the previous period {a j t-1 } j∈Bi (B i is the set of neighbors of agent i augmented by i) and the agent id i which identifies the agent's network position. To maintain a constant length input vector, unobserved actions are encoded as -1. Thus, the corresponding DQN is a function

Q : R × {-1, 0, 1} |N | × N × {0, 1} → R.

In the agent takeover attack, the attacker is represented by a DQN. This DQN is distinct from the citizen DQN. At the beginning of each episode the attacker chooses a citizen uniformly at random from N and acts on his behalf. In period t, the attacker observes θ without noise, his and the actions of his neighbors in the previous period {a j t-1 } j∈Bi and the agent id i. Thus, the corresponding DQN is a function

Q : {0, 1} × {-1, 0, 1} |N | × N × {0, 1} → R.

In the biased signal attack, the attacker is not represented by a DQN. Instead, the biases are hand-tuned. At the beginning of each episode the attacker chooses a set of citizens uniformly at random from N and delivers biased signals to them as outlined above.

Measuring information aggregation Intuitively, agents should be able learn to extract information from their private signals and their neighbors' actions, such that at the end of an episode their expected reward (E[r i T ]) should be significantly higher than at the beginning of an episode (E[r i 0 ]). That is, agents should aggregate information over time. In the following, we give a formal definition of information aggregation for finite N and T that is useful in our case.

Definition 1 (Information aggregation). Let F = {s i } i∈N denote the set of all realized signals in a given episode. Let θ = argmax θ P[θ | F] denote the maximum-a-posteriori estimator of θ given F. Then information is aggregated if for all i ∈ N , we have a i T = θ.

That is, under this definition information is aggregated if agents act as if they had seen all private signals. In reality, it is not reasonable to expect perfect information aggregation with actions learned via independent Q-learning. We therefore define the following measure of information aggregation.

Definition 2 (Accuracy of information aggregation). The accuracy of information aggregation at time step In the following we will simply refer to Ât as the accuracy.

t is A t = E[1{a i t = θ}]. A useful benchmark to compare A t to is of course E[1{ θ = θ}]. Another useful benchmark is E[1{ θ =

Heuristic benchmark We compare the accuracy of the DQN against the DeGroot heuristic, De-Groot (1974) . Under this heuristic information aggregation rule at t = 0, we set a i 0 = θ(s i ). In each subsequent period actions are set to a t = U a t-1 , where a t is the vector of actions taken by citizens and U = D -1 (I + M ), where I is the identity matrix, M is the graph adjacency matrix and D is a diagonal matrix with entries D ii being the number of neighbors of node i plus one (i.e. a self loop). Information is therefore aggregated through repeated averaging of past actions.

4. EXPERIMENTS:

Our objective is to improve our understanding of the propagation of fake news in social networks and discover potential interventions to curb its effectiveness. This goal translates into three main questions. (i) What determines the effectiveness of the attacker in reducing the accuracy of information aggregation? In particular, how does the network structure and the choice of the attacked agent affect attack effectiveness? (ii) How well can agents learn adapt to the presence of an attacker? (iii) When attacker and citizen learn simultaneously (agent takeover attack), how do policies change over time? To study these questions we set up a series of training and testing scenarios.

Training scenarios: We implement the following training scenarios. (i) Baseline: We train citizens in the absence of any attack, i.e. all citizens receive unbiased signals and no citizen is taken over by the attacker. We train baseline models for the Barabasi-Albert, clustered and balanced graphs. This scenario establishes a benchmark for the ability of IQL agents to learn to aggregate information in the absence of attack. (ii) Biased signal attack: We train citizens in the presence of a biased signal attack where a single, randomly chosen citizen receives a biased signal (β = 3). We train this model for the Barabasi-Albert graph only. This scenario allows us to evaluate whether citizens can learn to adapt to the presence of a biased signal attack and thereby mitigate the adverse effects of the attack. (iii) Agent takeover attack: We train citizens and attacker simultaneously. We train this model for the Barabasi-Albert graph only. This scenario allows us to understand the effectiveness of attack as it co-evolves with the citizen's effort to mitigate the attacker's ability to manipulate citizens. Note that IQL may not converge to stable policies in such a game. It is interesting however, to understand the action dynamics that emerge under IQL as they can help us understand the coevolution of fake news attack and defense in real social networks. (iv) Random action attack: We train citizens in the presence of an attacker that selects a single citizen uniformly at random and then, in each period, picks an action uniformly at random. We train this model for the Barabasi-Albert graph only. This scenario serves as a benchmark for the agent takeover attack. The accuracy under the random action attack should exceed the accuracy under the agent takeover attack.

Testing scenarios: Given the models obtained for these different training scenarios we conduct a number of testing scenarios. First, for each attack training scenario, we evaluate accuracy, as defined above, both in the presence of the corresponding attack (as trained) and in the absence of any attack. For the baseline scenario we consider two testing scenarios in addition to testing in the absence of any attack (as trained). First, we evaluate accuracy in the baseline model for the Barabasi-Albert graph in the presence of a biased signal attack with a single attacked agent and β = 3. Second, we evaluate accuracy in the baseline model for the clustered and balanced graphs under two biased signal attack scenarios. In the focused scenario, a single, randomly chosen citizen receives a biased signal with β i = 3. In the spread scenario, two, randomly chosen citizens receive biased signals with β i = 1.5 each.

5. RESULTS:

Our experiments yield four main results, which we will discuss in turn.

Baseline information aggregation::

In the absence of attack, citizens learn to aggregate information with an accuracy close to the optimal benchmarks E[1{ θ = θ}] as t → T and E[1{ θ = θ}] for t = 0. Our method's accuracy also substantially outperforms the accuracy achieved by DeGroot information aggregation heuristic. This highlights the appropriateness of IQL to solve the standard information aggregation game. In the presence of an attack, information aggregation is severely disrupted. These results are illustrated in Figure 2 (A) . 

Determinants of effectiveness of attack::

Our experiments allow us to study the following determinants of attack effectiveness: network position of the attacked agent, signal strength of the attacked agent, network structure (clustered vs balanced) and distribution of bias across agents (focus vs spread). Let us consider each in turn.

To study the effect of network position, we compute the accuracy in the final time step (t = T ) conditional on the network position of the attacked agent when evaluating the baseline model under the biased signal attack in the Barabasi-Albert network. Denote this conditional accuracy by ÂT (i), where i is the attacked agent. Let ÂT denote the baseline accuracy in the absence of attack. We then define the decline in accuracy relative to baseline as ∆A(i) = ÂT -ÂT (i). We plot ∆A(i) as a function of i in Figure 2 (C). It is clear that there is substantial heterogeneity in the attack effectiveness. As the tight correlation between ∆A(i) and the degree of i in Figure 2 (C) shows, the heterogeneity in ∆A(i) arises from the network structure which implicitly gives more influence to those nodes with more neighbors (higher degree). If an attacker is able to target the biased signal to such highly connected nodes, his attack will be more effective.

Next we study the effect of the signal strength of attacked agent. Again, we restrict ourselves to evaluating the baseline model under the biased signal attack in the Barabasi-Albert network. A natural measure of a signal's strength, or informativeness, is the absolute value of the log likelihood ratio of the two states of θ conditional on the signal. Let f (θ | s) denote the posterior distribution over θ given the signal s. Then, the signal strength is given by L

(s) = | log[f (θ = 1 | s)/f (θ = 0 | s)]|.

Let L k denote the kth decile of the empirical distribution of signal strengths for a particular experiment. We define the conditional accuracy Â(L k ) as the average accuracy in the final time step conditional on the attacked agent's signal strength lying in

[L k , L k+1 ]. ∆A(L k ) is defined analogously to ∆A(i). We plot ∆A(L k ) against L k in Figure 2 (D).

There exists a strong negative correlation between attack effectiveness as measured by ∆A(L k ) and the attacked agent's signal strength. This is intuitive. If an agent has received a strong private signal in support of some value of θ, a larger bias will be required to convince him of the contrary. Thus, if an attacker can target the biased signal to agents with weak private signals, his attack will be more effective. Lastly, we consider the case of clustered vs rewired/balanced networks and spread our vs focused attacks. This analysis is done evaluating the baseline model under the biased signal attack in the clustered and balanced networks. A number of results are worth noting. As can be seen in Figure 2 (E), for each scenario (baseline/no attack, focus and spread attacks) accuracy in the balanced network exceeds accuracy in the clustered network. This suggests that information aggregation is more effective in the balanced network. This is intuitive as the balanced networks has a shorter maximum path length between any two nodes thereby allowing information to "propagate" faster between nodes. In Figure 2 (E), we can also see that variation in the accuracy between training runs with different seeds (each run corresponds to one marker for a particular scenario) is smaller for the balanced network. This suggests that in the balanced network learning good policies is an easier task than in the clustered network. We can also see from Figure 2 (E) that spread attacks are more effective both in the clustered and balanced networks.

To summarize, targeting highly connected agents in the network or those with weak private signals, makes biased signal attacks more effective. Clustered networks are more susceptible to attack than balanced networks and spread attacks are more effective than focused ones.

Adapting to attacks: So far we have evaluated models trained in the absence of attack. However, one can expect that users of social networks will adapt over time to the presence of fake news. We therefore investigate to what extent citizens can learn to mitigate the effect of an attacker. For this purpose we train citizens in the presence of a biased signal attacker, a random attacker and an agent takeover attacker for the Barabasi-Albert network. We then evaluate the accuracy of each of these models in the absence an attack and in the presence of the attack under which they were trained. We summarize our results in Table 1 .

Let us first contrast the accuracy of the baseline model (trained without attack) under a biased signal attacker with the accuracy of a model trained under this attack scenario. When trained in the presence of an attacker the test accuracy under attack increases from 0.78 to 0.84 relative to the baseline. This can also be seen in Figure 2 (B). We conclude that agents can indeed learn to adapt to the presence of fake news. However, this adaptation comes at a cost. When trained in the presence of an attacker the test accuracy without attack decreases from 0.92 to 0.88 relative to the baseline. We conjecture that this is because agents learn to trust strong private signals less and are less likely to follow their neighbors' actions. A similar picture can be seen for the random attacker and the agent takeover attacker. To summarize, agents can learn to adapt to the presence of an attacker which reduces attack effectiveness. However, adaptation comes at a cost. If trained under attack and tested in the absence of attack, accuracy is lower than in the baseline which is trained without attack.

Dynamics of attack and defense::

We now take a closer look at the agent takeover attack. Recall that in this scenario, a separate DQN agent is trained to minimize the total reward of the citizens by acting on behalf of a randomly chosen agent. This attacker knows θ and must then choose an action to convince the citizens of the opposite. Simply choosing the opposite action would be a naive strategy that is likely to be detected quickly by the citizens as they learn in the presence of this strategy. Always choosing a random action in every period, as the random attacker does, is also quite easily detectable by the citizens. This intuition is confirmed by results reported in Table 1 and Figure 3 .

From Table 1 we see that the trained attacker clearly "outperforms" (i.e. more effective attack) the random action attack benchmark. The bottom panel of Figure 3 (B) shows the accuracy of the attacker in the first time step as a simple summary statistic of its strategy. The accuracy is either 0.5 indicating that the attacker "randomizes" or 0 indicating that the attacker "lies". While Q-learning only allows for deterministic policies the attacker can learn to lie if the attacked agent is in one partition of N and be truthful if the attacked agent is in another partition. Since the attacked agent is chosen at random, this strategy amounts to a randomized attack strategy.

Note further, that neither the attacker's nor the citizens' policy converges as can been seen in the highly variable reward and unstable TD-error in Figure 3 (A) . Indeed, as can be seen in the bottom panel of Figure 3 (B) , the attacker's policy fluctuates between randomization and lying between episodes. These fluctuations can be seen as irregular "cycles" in the policy space of the attacker and the citizen. One such cycle in Figure 3 (B) shows the attacker switching to a lying strategy, while the citizens lose "trust" in their private signal (they nearly ignore it in the first time step). The result of these two events coming together can lead to substantial, temporary declines in accuracy.

We don't claim that these dynamics are accurate representations of the learning dynamics between users of social networks and pushers of fake news. However, we consider the failure of IQL to converge and the irregular dynamics that emerge more of a feature than a bug. Indeed these dynamics suggest two interesting, qualitative take aways. First, attackers and citizens are unlikely to converge to stable policies in real life. We should expect a continuing cycle of adaptation with periods of a high prevalence of fake news followed by periods of low prevalence. Second, the co-evolution of attackers and citizens can have unexpected effects on the citizens' ability to aggregate information in the social network. As we observe in Figure 3 (B) , the presence of neighbors that do not act in accordance with their private signals (i.e. the attacked agents) can lead citizens to learn to mistrust their own private signals.

6. CONCLUSION AND DISCUSSION:

The deliberate manipulation of the public's perception of facts via fake news, in particular on social networks, has become a growing concern for policy makers and technology companies alike. Despite its importance, little is understood about the spread of fake news on social networks. This is in large part due to the technical difficulties involved in studying complex decision making behavior on social networks. We seek to close this gap by developing the (to the best of our knowledge) first framework for the study fake news on social networks that is theoretically grounded yet com-putationally tractable and flexible. We achieve this by (i) extending a standard game of information aggregation to accommodate fake news and (ii) applying state-of-the-art deep multi-agent reinforcement learning to solve the game.

Our findings suggest a number of interventions that could contribute to making fake news attacks less effective. First, if agents are made aware of the presence of fake news, they can learn to adapt and mitigate its effect. However, this adaptation is likely to be accompanied by evolving attack strategies, so any adaptation will not be lead to permanent mitigation. In addition, the adaptation is likely to harm information aggregation in the absence of fake news. Second, keeping private information on social networks private, can make it harder for attackers to condition their attacks on network position and general informedness (as the real world analogue of signal strength). This should reduce attack effectiveness. Third, encouraging well balanced social networks can improve information aggregation in general and make fake news attacks less effective in particular.

Our current approach has two main shortcomings. First, it does not scale well to larger populations of agents since it requires the entire social network to be trained simultaneously. One way forward is to train agents in sub-graphs feeding them an embedding of the graph, see for example Gilmer et al. (2017) , and then composing larger graphs at test time. The second shortcoming is the use of IQL in the agent takeover attack scenario. While the lack of convergence leads to interesting qualitative insights, MARL methods that encourage convergence of policies would be desirable here, at least as benchmarks. One possible way to achieve this would be to apply stable opponent shaping to citizen and attacker, see Letcher et al. (2018) .

:

We restrict our analysis to the case when claims are factual statements that can be objectively verified as true or false. Of course this prevents us from studying the dynamics of opinions. However, since we focus on the effect of fake news on the social network, restricting our analysis to factual statements is natural.

Biased signals can be thought of as targeted ads to users of the social network. Indeed, it been shown that fake news attacks have taken exactly this form, seeChiou & Tucker (2018). A fixed bias budget corresponds roughly to an attacker with a fixed budget for targeted social media ads.

Figures and Tables:

Figure 1: Left: Instance of a Barabsi-Albert random network with |N | = 10 that was used in our analyses (unless otherwise stated). Middle: stereotypical clustered network with |N | = 12 with three (nc = 3) fully connected clusters of four nodes (sc = 4) each. Right: the clustered network in balanced form produced by random rewiring of link pairs (preserves degree distribution). These networks were used in our analysis of the effectiveness of information aggregation and attack (i) in clustered vs. balanced networks and (ii) for spread out vs. focused attack strategies.

θ}], where θ = argmax θ P[θ | s i ] is the MAP estimator given only a single private signal. A T can never exceed E[1{ θ = θ}] and A 1 can never exceed E[1{ θ = θ}] and of course E[1{ θ = θ}] > E[1{ θ = θ}] provided |N | > 1. Once we have trained the agents via Q-learning it is easy to estimate A t by evaluating a set of batches of games B and computing Ât = 1/|B|1/|N | b∈B,i∈N 1{a i t,b = θ}, where a i t,b is the action taken by agent i in batch b ∈ B.

Figure 2: (A-B): Information aggregation over time in Barabasi-Albert graph. (A) Baseline scenario. Under the attack test scenario, a single agent receives a biased signal. (B) Biased signal attack scenario. The upper and lower dashed lines corresponds to the benchmarks E[1{ θ = θ}] and E[1{ θ = θ}] respectively. Square markers correspond to the accuracy achieved by the De Groot information aggration heuristic in the baseline and biased signal attack scenarios respectively. (C-D) Agents are trained in the absence of any attacker on a Barabasi-Albert graph. (C) Top: Average decline in accuracy conditional on agent id which determines network position. Bottom: degree of attacked agent. (D) Effectiveness of conditioning on attacked agent signal strength. (E) Agents trained in the absence of any attacker for clustered (left) and rewired / balanced (right). Base: no attack. Focus: a single, randomly chosen agent receives a strong biased signal (β i = 3). Spread: two, randomly chosen agents receive weak biased signals (β i = 1.5). Blue markers represent training runs with different seeds for a given network and attack scenario. Orange markers are averages over the different training runs. Error bars computed over outcomes with different random seeds but fixed neural network weights are too small to be visible on the graphs.

Figure 3: Dynamics of co-evolving attacker and citizen agents on a Barabsi-Albert graph. (A) Red lines are exponentially weighted moving averages. Top: accuracy in the last time step (t = T = 10) over episodes. Bottom: TD error of Q learning. (B) Close up of accuracy and actions of citizens and attacker early on during training averaged over multiple batches. Top: citizen accuracy in the last time step (t = T = 10). Bottom: citizen and attacker accuracy in the first time step (t = T = 0).

Test accuracy of agents in the final time step (t = T = 10) for different training and testing scenarios.All scenarios were run with the Barabasi-Albert graph. At test time without attack all agents receive unbiased signals and no agent is attacked. At test time with attack agents are exposed to the attack scenario they were trained under (except in the baseline case where attack is biased signal with β = 3).

Table content: <table><tr><td/><td colspan="2">Testing accuracy (t = T )</td></tr><tr><td>Training scenario</td><td colspan="2">Without attack With Attack</td></tr><tr><td colspan="2">Baseline (train: no attack, test: hand-tuned β = 3) 0.926</td><td>0.784</td></tr><tr><td>Biased signal attack (β = 3)</td><td>0.882</td><td>0.849</td></tr><tr><td>Random action attack</td><td>0.909</td><td>0.908</td></tr><tr><td>Agent takeover attack</td><td>0.863</td><td>0.844</td></tr></table>

Peer review (iclr2020):
In this work, the authors aim to solve the problem of fake news detection in social media. The proposed method is built upon a multi-agent reinforcement learning method. Although the problem of fake news detection in social media has been extensively studied and many method including deep learning have been investigated to help solve the problem, it is relatively novel to use multi-agent reinforcement learning in this field. The proposed method is based on traditional multi-agent deep reinforcement learning approach and the authors extend the conventional framework by introducing the role of attacking agents - agents that can spread biased information or even take over the stance of regular users. The paper has been well written and necessary details for reproducing the experimental results have been provided with a link to the code repository. However, a major concern of mine is the contribution of novelty of the manuscript. The main contribution of novelty of the work, as claimed by the authors, is that they come up with a practical solution for fake news detection with deep reinforcement learning. Most research efforts have been focusing on detecting fake and deep-fake content while very few pay attention to utilizing machine learning in learning a best action/practice. This is because the root cause of the widespread of fake news is quite complicated. Besides those who intentionally create and spread fake news and biased content, the innocent users* major problem is how to quickly identify fake news from massive amount of information flows. The idea that building a practical fake news prevention solution by using multi-agent reinforcement learning seems to have underestimated the complexity of the misinformation challenge. For example, three high-level suggestions/solutions proposed in the manuscript include social network users should be more aware of the presence of fake news, keeping private information private on social networks, and encouraging well balanced social network structures. My question is that how we can apply these solutions in the real world? Therefore, I think this piece of work is more theoretical rather than practical. In terms of the technical part, the authors propose to introduce agents for fake news and biased information. The technical solution is solely based on multi-agent reinforcement learning and the extension is straightforward. The assumption that there is only one kind of role for fake news dissemination in social networks again underestimates the complexity of the problem in the real world. E.g., there are users who intentionally create fake news in social networks, and also users who are not aware of a piece of information being fake and yet still deeply believe what they spread is true. I would suggest the authors to find more related work in the field of information diffusion, where researchers have long been focusing on competitive information propagation in social networks with multiple parties (such as political campaign and word-of-mouth social marketing). In conclusion, I think the work is well-written and quite interesting - solving an emerging and important problem from a new perspective. However, both the hypothesis and the technical solution are lacking enough contributions of novelty. I would like to suggest the authors either make the solution actually practical or focus more on the theoretical part of competitive information diffusion in social networks.