Modeling Fake News in Social Networks with Deep Multi-Agent Reinforcement Learning
Authors: Christoph Aymanns
Matthias Weber
Co-Pierre Georg
Jakob Foerster
Venue:  (2020)
Keywords: deep multi-agent reinforcement learning
fake news
social networks
information aggregation

Abstract: We develop a practical and flexible computational model of fake news on social networks in which agents act according to learned best response functions. We achieve this by extending an information aggregation game to allow for fake news and by representing agents as recurrent deep Q-networks (DQN) trained by independent Q-learning. In the game, agents repeatedly guess whether a claim is true or false taking into account an informative private signal and observations of actions of their neighbors on the social network in the previous period. We incorporate fake news into the model by adding an adversarial agent, the attacker, that either provides biased private signals to or takes over a subset of agents. The attacker can follow either a hand-tuned or trained policy. Our model allows us to tackle questions that are analytically intractable in fully rational models, while ensuring that agents follow reasonable best response functions. Our results highlight the importance of awareness, privacy and social connectivity in curbing the adverse effects of fake news.


Under review as a conference paper at ICLR 2020
MODELING FAKE NEWS IN SOCIAL NETWORKS WITH DEEP MULTI-AGENT REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Over the last few years deep multi-agent reinforcement learning (DMARL) has become an increasingly active area of research with hundreds of papers submit- ted to top machine learning conferences every year. However, so far there have been few real world use cases that benefited from the progress in the field. We use DMARL to develop a flexible computational model of fake news social networks in which agents act according to Iearned best response functions. We achieve this by extending information aggregation game to allow for fake news and by representing agents as recurrent deep Q-networks (DQN). In the game, agents repeatedly guess whether claim is true false taking into account informative private signal and observations of actions of their neighbors on the social network in the previous period. We incorporate fake news into the model by adding an adversarial agent, the attacker, that either provides biased private signals to, or takes over, a subset of agents. The attacker can follow either a hand-tuned or trained policy. Our model allows us to tackle questions that are analytically intractable fully rational models, while ensuring that agents follow reasonable best response functions. Our results highlight the importance of aware- ness, privacy and social connectivity in curbing the adverse effects of fake news and open an entire new real world application area for DMARL.
INTRODUCTION
Problem setting At least since the 2016 US presidential election, fake news on social networks, aimed manipulating the users' perception of facts, has been recognized societies. Yet, to date little is understood about how fake news spread on social networks, when a piece of fake news is effective in swaying public opinion and what interventions might be successful in mitigating the effect of fake news. Our lack of understanding of this important phenomenon is in large part due the difficulty of modeling complex decision making processes on social networks. To tackle this issue, we extend a standard game of information aggregation on social network, see Mossel et al.) (2015), to accommodate fake news and solve this, analytically intractable, game using deep multi-agent reinforcement learning. This approach is flexible, computationally tractable, and allows us to work towards answering the questions raised above.
major issue in open
Related work Our work relates two main strands of literature: (i) models of information aggre- gation in social networks and (ii) deep multi-agent reinforcement learning as a tool to solve games. The standard information aggregation game, outlined in Section 2. involves agents (repeatedly) guessing an unknown state of the world taking into account an informative private signal and ob- servations of actions of their neighbors on the social network in the previous period. Note that this model involves no fake news. There are two main approaches to studying this game. In the rational agent approach, agents' decision are best responses and the analysis focuses on the Nash equilibria of the game. In an important contribution, MMosell et al. (2015) show that under certain conditions on the social network, in the limit T + oo (number of periods) and n + oc (number of agents), agents will converge and will agree the correct state of the world. This approach is not tractable for finite T and n and therefore does not allow, for example, the study of transient behavior. An al- ternative approach to studying the game of information aggregation is heuristic, see Golub & Sadler (2016) for a review of many important contributions. While heuristic approaches are more flexible to changes in the model setup and allow the study of transient behavior, agent strategies can be too

---

Under review as a conference paper at ICLR 2020
simplistic. For example, agents may treat repeated observations of neighbor's guess as conveying additional information, even when this is not the case (this occurs for example when this neighbor's s only information is his private signal). Furthermore, heuristic behavior does not adapt to changing circumstances and would therefore not allow us to study how agent behavior changes in the pres- ence of fake news. Our deep multi-agent reinforcement learning approach combines both worlds. Our approach is flexible since it does not require the model to be analytically tractable and finds near-optimal policies through reinforcement learning.
Our work is also related to studies of competitive information diffusion, see Alon el al. (2010); Etesami & Başar (2014). However, while the literature on competitive information diffusion focuses on the strategic seeding of a pre-determined diffusion process, our work focuses on modeling the information diffusion process itself through the agents' learned actions.
Deep multi-agent reinforcement learning (DMARL) has been applied to solve games of differ- ent structure. Policies or value functions are approximated by deep neural nets and trained via backpropagation. Some recent examples are Letcher et al. (2018); Foerster et al. (2018); Lerer & Peysakhovich (2017): [Balduzii et al. (2018). Often the focus is on methods that allow agents to converge particular, socially desirable policy, such as cooperation in social dilemmas without changing the payout structure. Since this is not an issue for most of our analyses, we make use of Independent Q-learning, see Tampuu al. (2077). However, for some of more advanced anal- yses, convergence does become an issue and applying more advanced DMARL methods provides a promising avenue of future research.
Contribution To the best of our knowledge, we are the first to develop a flexible computational model of fake news on social networks in which agents act according to learned best response functions. We achieve this by extending a standard information aggregation model to allow for fake news and by representing agents as recurrent deep Q-networks (DQN). We incorporate fake news into the model by adding an adversarial agent, the attacker, that either provides biased private signals to takes over subset of agents. We conduct number of experiments main questions. (i) What determines the effectiveness of the attacker in reducing the accuracy of information aggregation? (ii) How well can agents learn adapt to the presence of an attacker? (iii) When attacker and agents learn simultaneously, how do policies evolve over time? Our results suggest three potential interventions to curb the effectiveness of fake news social networks. First, "vaccinate" users of social networks by making them aware of the presence of fake news. Second, keep private information on social networks private so that attackers cannot target poorly informed or well connected users. Third, encourage the formation of "balanced" rather than clustered or fragmented social networks, so that information can "flow" effectively across the network.
answer three
Lastly, we contribute to the MARL literature by providing an important and concrete real-world application of methods that date have found limited practical application.
2 BACKGROUND
2.1 AN INFORMATION AGGREGATION GAME ON SOCIAL NETWORK
We consider a sequential game of information aggregation over T periods indexed by L 1,..., T with a set of agents N, see Mossel et al. 2015); Golub Sadler (2016) for details. Agents interact via fixed, possibly directed, graph NVN E). where E denotes the set of edges connecting the nodes N; see Figure [ for an illustration of the networks used in this paper. For concreteness, one can think of the agents users of social network. such as Twitter. A directed link from agent A to agent B would then imply that A follows B on Twitter.
Agents presented with claim that is either true or false. Again, for concreteness, can think of the claim as a factual statement encountered in a piece of news that has been shared among the agents on the social network [ True claims are denoted as 0 and false claims as 0 = 0. True and false claims occur with equal probability; that is, the unconditional probability of a claim being
We restrict our analysis to the case when claims are factual statements that can be objectively verified as true or false. Of course this prevents us from studying the dynamics of opinions. However, since we focus on the effect of fake news on the social network, restricting our analysis to factual statements is natural.
2

---

Under review as a conference paper at ICLR 2020
Bar Albert graph
Clustered graph
Rewired form clustered graphh
10
Figure 1: Left: Instance a Barabsi- Albert random network with |N|
that was used in our analyses
(unless otherwise stated). Middle: stereotypical clustered network with [N] = 12 with three (ne = 3) fully connected clusters of four nodes (s, = 4) each. Right: the clustered network in balanced form produced by random rewiring of link pairs (preserves degree distribution). These networks were used in our analysis of the effectiveness of information aggregation and attack (i) in clustered vs. balanced networks and (ii) for spread out vs. focused attack strategies.
true false is P(O - 0) P(O - 1/2. This choice simplifies our analysis without affecting any of our qualitative results. Agents receive private signals 8 ~ Fg once at the beginning of an episode. In the baseline F6 is such that private signals are always informative but noisy. In each period agents choose a binary action aj E {0, 1}. Agents observe the actions of their neighbors on the graph from the previous period. At the end of the episode, agents are rewarded for actions that matched the veracity of the claim presented. For given claim 0 and sequence of actions {at), an agent's total discounted reward is ri = Σ 75''"1(6 {af = 0).
Note that in this game, for 1 < T, it is not necessarily optimal to choose the action that matches the maximum of the posterior likelihood of 0. This is because an agent wants prevent others from simply copying its action, since that reveals no information about the others' private signal. Injecting noise into actions may therefore be beneficial. Despite these concerns, it can be shown that, under certain conditions on the network, in the limit T + co (number of periods) and n - oc (number of agents), agents will converge and will agree on the correct 6, see Mossel al. (0055)
2.2 MULTI-AGENT REINFORCEMENT LEARNING: INDEPENDENT Q-LEARNING
In principle, reinforcement learning (RL) can be used to learn the agents' policy (or response func- tions) in any game. Many methods have been proposed to learn reasonable best response functions in various multi-agent reinforcement learning (MARL) settings, see Foerster et al. (0066); Letcher et al. (2018); Lerer& & Peysakhovich (2017). In this paper, the agents' response functions will be learned via independent Q-learning (IQL) and Q functions will be approximated by deep recurrent neural networks, see Tampuu et al. 2017 Foerster et al. (2016). We relegate discussion of the RL. and IQL framework to Appendix A.
3 METHODS
Graphs We consider three types of undirected graphs, see Figure for illustration. An instance of a Barabasi-Alhert random graph with |N| W|=10 10, see Barabási & Albert (1999); a clustered graph with three clusters (n, = 3) of fully connected nodes with four nodes each (s. 4) such that |N| = 12: and a balanced graph (N) = 12) that is obtained by randomly rewiring the clustered graph. Random rewiring involves repeatedly selecting two edges uniformly random and swapping the terminal nodes of these edges. This preserves the degree sequence, i.c. the number of neighbors of each node, but removes the clusters such that the rewired graph is "balanced", The Barabasi- Albert random graph is often used to model social networks and is therefore natural choice for our application. The clustered graph a stereotypical example of a social network which is comprised of cliques that are weakly connected. It thereby represents more "fragmented" or "polarized" social network. The balanced graph removes these clusters from the graph while maintaining the degree sequence and can thus help isolate the effect of clusters on information aggregation.
3

---

Under review as a conference paper at ICLR 2020
Fake news We extend the model of information aggregation in order to study the effects of fake news on information aggregation. We introduce an adversary, the so called attacker, who wants to reduce the other agents' total reward E, 7;- To disambiguate between the attacker and other agents, we often refer to these other agents as citizens. The attacker wants to persuade citizens to support false claims and dissuade them from supporting true claims. The attacker can achieve this in one of two ways. First, in the biased signal attack, he can distribute a fixed budget of bias 3 = >e 3 across a subset of the agents such that citizen i's private signal is drawn from sf ~ Fo(B:) We choose s¹ ~ (e + (1 20). We set o² 1 throughout. In this case the choice of J is hand-tuned, i.e. set by the experimenter and not optimized. Second, in the agent takeover attack, he can directly take over an agent and act on his behalf. this case the attacker optimizes his attack policy by IQI..
Learning to aggregate information We hold the network fixed across episodes. At the beginning of each episode 0 and all private signals are drawn at random according to the distributions given above. For citizens we train a single DQN. In period t, the citizen observes the episode private signal s'. his and the actions of his neighbors in the previous period {a'_1}jE (B; is the set of neighbors of agent augmented by i) and the agent id which identifies the agent's s network position. To maintain a constant length input vector, unobserved actions are encoded as -. Thus, the corresponding DQN is a function Q R x 1-,,0, x N x {0, 1 + R.
In the agent takeover attack, the attacker is represented by DQN. This DQN is distinct from the citizen DQN. At the beginning of each episode the attacker chooses citizen uniformly at random from N and acts on his behalf. In period f, the attacker observes @ without noise, his and the actions of his neighbors in the previous period {ai=)heR, and the agent id i. Thus, the corresponding DQN is a function Q: : {0, 13 x (-,,0, NI x N x {0, 10 R. In the biased signal attack, the attacker not represented by a DQN. Instead, the biases are hand-tuned. At the beginning of each episode the attacker chooses a set of citizens uniformly at random from N and delivers biased signals to them as outlined above.
Measuring information aggregation Intuitively, agents should be able learn extract informa- tion from their private signals and their neighbors' actions, such that at the end of an episode their expected reward (EIr+) should be significantly higher than at the beginning of an episode (E(r)) That is, agents should aggregate information over time. In the following, we give formal definition of information aggregation for finite N and T that is useful in our case.
Definition 1 (Information aggregation). Let F = {s JieN denote the set of all realized signals in a given episode. Let Ô = argmax, FF] denote the maximum-a-posterin estimator of 0 given F. Then information is aggregated if for all E N, we have atp- = Ó.
That is, under this definition information is aggregated if agents act as if they had seen all private signals. In reality, it is not reasonable to expect perfect information aggregation with actions learned via independent Q-learnin. We therefore define the following measure of information aggregation.
Definition 2 (Accuracy of information aggregation). The accuracy of information aggregation at time step is A, = (aj = 0}}.
A useful benchmark to compare A; to is of course E 0H}. Another useful benchmark is ( = 0}1. where = argmax, s²] is the MAP estimator given only a single pri- vate signal. Ar can never exceed E|1 (Ø = 0}] and A₁ can never exceed E[ (0 = 0}] and of course E[1 ( 01] E 0} provided |N| 1. Once we have trained the agents via Q-learning it is easy to estimate A, by evaluating a set of batches of games B and computing A, = 1/ / EiERiiEN 1 Iaa, = 0). where at,, is the action taken by agent in batch E B.
In the following we will simply refer to A, as the accuracy.
2 Biased signals can be thought of as targeted ads to users of the social network. Indeed, it been shown that fake news attacks have taken exactly this form, see Chiou & Tucker (2018). A fixed hias budget corresponds roughly attacker with fixed budget for targeted social media ads.
4

---

Under review as a conference paper at ICLR 2020
Heuristic benchmark We compare the accuracy of the DQN against the DeGroot heuristic, De- Groot 1974). Under this heuristic information aggregation rule at t = 0, we set aj O(s"). In each subsequent period actions are set to a = Ua,-1, where 04 is the vector of actions taken by citizens and U D (7 + M). where I is the identity matrix, M is the graph adjacency matrix and D is diagonal matrix with entries D being the number of neighbors of node plus one (i.e. a self loop).
Information is therefore aggregated through repeated averaging of past actions.
EXPERIMENTS
Our objective is improve understanding of the propagation of fake news in social networks and discover potential interventions to curb its effectiveness. This goal translates into three main questions. (i) What determines the effectiveness of the attacker in reducing the accuracy of informa- tion aggregation? In particular, how does the network structure and the choice of the attacked agent affect attack effectiveness? (ii) How well can agents learn adapt to the presence of attacker? (iii) When attacker and citizen learn simultancously (agent takeover attack), how do policies change over time? To study these questions we set up series of training and testing scenarios.
Training scenarios: We implement the following training scenarios. (i) Baseline: We train citi- zens in the absence of any attack, i.e. all citizens receive unbiased signals and no citizen taken over by the attacker. We train baseline models for the Barabasi-Albert, clustered and balanced graphs. This scenario establishes a benchmark for the ability of IQL agents to learn to aggregate informa- tion in the absence of attack. (ii) Biased signal attack: We train citizens in the presence of biased signal attack where a single, randomly chosen citizen receives a biased signal (B = 3). We train this model for the Barabasi-Albert graph only. This scenario allows to evaluate whether citizens can learn to adapt to the presence of biased signal attack and thereby mitigate the adverse effects of the attack. (iii) Agent takeover attack: We train citizens and attacker simultaneously. We train this model for the Barabasi-Albert graph only. This scenario allows us to understand the effectiveness of attack as it co-evolves with the citizen's effort to mitigate the attacker's ability to manipulate cit- izens. Note that IQL may not converge to stable policies in such a game. It is interesting however, to understand the action dynamics that emerge under IQL as they can help us understand the co- evolution of fake news attack and defense in real social networks. (iv) Random action attack: We train citizens in the presence of an attacker that selects a single citizen uniformly at random and then, in each period, picks an action uniformly at random. We train this model for the Barabasi-Albert graph only. This scenario serves as a benchmark for the agent takeover attack. The accuracy under the random action attack should exceed the accuracy under the agent takeover attack.
Testing scenarios: Given the models obtained for these different training scenarios we conduct a number of testing scenarios. First, for each attack training scenario, we evaluate accuracy, as defined above, both in the presence of the corresponding attack (as trained) and in the absence of any attack. For the baseline scenario we consider two testing scenarios in addition testing in the absence of any attack (as trained). First, we evaluate accuracy in the baseline model for the Barabasi-Albert graph in the presence of biased signal attack with single attacked agent and / 3. Second, we evaluate accuracy in the baseline model for the clustered and balanced graphs under two biased signal attack scenarios. In the focused scenario, a single, randomly chosen citizen receives a biased signal with 8 3. In the spread scenario, two, randomly chosen citizens receive biased signals with 3r = 1.5 each.
RESULTS
Our experiments yield four main results, which we will discuss in turn.
Baseline information aggregation: In the absence of attack, citizens learn to aggregate informa- tion with an accuracy close to the optimal benchmarks [[ = 0}] as t + T and [[ 0Jl for = O. Our method's 's accuracy also substantially outperforms the accuracy achieved by DeGroot information aggregation heuristic. This highlights the appropriateness of IQL to solve the standard information aggregation game. In the presence of an attack, information aggregation is severely disrupted. These results are illustrated in Figure 2 (A).
5

---

Under review as a conference paper at ICLR 2020
A
1.0
attack
B
1.0
Train: signal attack
Deciie in eccuraty
0.15
C
wadne @ 20375-
D
0.10-
0.22-
0.05
0..00
Accuracy
0.85
Accuracy
0.00
0.00
Signal
clustered
rewired
0.7
0.7
S Accurayyo
private signal optimal all signals optimal
private signal optimal
Degreer
0.65
all signals optimal
0.6
DeGroon: no attack
0.6
DeCront: attack
0.50
test: attack
test: attack
rest: bissed Gignal
test: biased signal
0.70
0
2
Time 6
0.5
2
6
8
Attacked agent
hane spreaed
focui spriaal
Time step
Figure 2: (A-B): Information aggregation over time in Barabasi-Ales Albert graph. (A) Baseline scenario. Under the attack test scenario, single agent receives a biased signal. (B) Biased signal attack scenario. The upper and lower dashed lines corresonns the benchmarks B[1 (Ō- 0}] and [[[ (Ō 0)] respectively. Square markers correspond the accuracy achieved by the De Groot information aggration heuristic the baseline and biased signal attack scenarios respectively. (C-D) Agents are trained in the absence of any attacker on a Barabasi-Albert graph. (C) Top: Average decline in accuracy conditional on agent id which determines network position. Bottom: degree of attacked agent. (D) Effectiveness of conditioning
attacked agent signal strength. clustered (left) and rewired balanced (right). Base: no attack. Focus: a single, randomly chosen agent receives strong biased signal (8' = 3). Spread: two, randomly chosen agents receive weak biased signals (3' = 1.5). Blue markers represent training rums with different seeds for given network and attack scenario. Orange markers averages over the different training runs. Frror bars computed over outcomes with different random seeds but fixed neural network weights are too small to be visible on the graphs.
(E) Agents trained in the absence of any attacker
Our experiments allow to study the following de- terminants attack effectiveness: network position of the attacked agent, signal strength of the attacked agent, network structure (clustered vs balanced) and distribution of bias across agents (fo- cus vs spread). Let us consider each in turn.
Determinants of effectiveness of attack:
To study the effect of network position, we compute the accuracy in the final time step (t T) conditional on the network position of the attacked agent when evaluating the baseline model under the biased signal attack in the Barabasi-Albert network. Denote this conditional accuracy by (i), where i is the attacked agent. Let Ar denote the baseline accuracy in the absence of attack. We then define the decline in accuracy relative to baseline as AA(i) = Ây as a function of in Figure 2 (C). clear that there is substantial heterogeneity in the attack effectiveness. As the tight correlation between AA(i) and the degree of i in Figure ( (C) shows, the heterogeneity in A Ai) arises from the network structure which implicitly gives more influence to those nodes with more neighbors (higher degree). If an attacker is able target the biased signal to such highly connected nodes, his attack will be more effective.
(i). We plot AA(i
Next study the effect of the signal strength of attacked agent. Again, restrict ourselves to evaluating the baseline model under the biased signal attack in the Barabasi-Albes network. A natural measure of signal's strength, informativeness, is the absolute value of the log likelihood ratio of the two states of 0 conditional on the signal. Let f(O s) denote the posterior distribution over 0 given the signal s. Then, the signal strength is given by L(s) = [log[/(0 Let La denote the kth decile of the empirical distribution of signal strengths for particular
s)
0 })]l experiment. We define the conditional accuracy Á(Lx) as the average accuracy in the final time step conditional on the attacked agent's signal strength lying in [Lx, Le+ AA(L)) defined analogously to AA(i). We plot AA(Ls) against La in Figure (D). There exists strong negative correlation between attack effectiveness as measured by A AA(Ls) and the attacked agent's 's signal strength. This is intuitive. If an agent has received strong private signal in support of some value of f, a larger bias will be required to convince him of the contrary. Thus, if an attacker can target the biased signal to agents with weak private signals, his attack will be more effective.

---

Under review as a conference paper at ICLR 2020
Test accuracy of agents in the final time step (t = T'= 10) for different training and testing scenarios. All scenarios were run with the Aerabasi-Albent graph. At test time without attack all agents receive unbiased signals and no agent is attacked. At test time with attack agents are exposed to the attack scenario they were trained under (except in the haseline case where attack is hiased signal with 3 3).
Table 1:
Testing accuracy (f (f=1 T)
Training scenario
Without attack
With Attack
Baseline (train: no attack, test: hand-tuned | 3) Biased signal attack (8 = 3)
0.926
0.784
0.882
0.849
Random action attack
0,909
0.908
Agent takeover attack
0.863
0.844
Lastly, we consider the case of clustered vs rewired/balanced networks and spread our vs focused attacks. This analysis is done evaluating the baseline model under the biased signal attack in the clustered and balanced networks. A number of results are worth noting. As can be seen in Figure 2 (E), for each scenario (baseline/no attack, focus and spread attacks) accuracy in the balanced network exceeds accuracy in the clustered network. This suggests that information aggregation is more effective in the balanced network. This is intuitive the balanced networks has shorter maximum path length between any two nodes thereby allowing information to "propagate" faster between nodes. In (E), we can also see that variation in the accuracy between training runs with different seeds (each run corresponds to one marker for particular scenario) is smaller for the balanced network. This suggests that in the balanced network learning good policies an easier task than in the clustered network. We can also see from Figure 2 (E) that spread attacks are more effective both in the clustered and balanced networks.
To summarize, targeting highly connected agents in the network or those with weak private signals, makes biased signal attacks more effective. Clustered networks are more susceptible to attack than balanced networks and spread attacks are more effective than focused ones.
Adapting attacks: So far we have evaluated models trained in the absence of attack. However, one can expect that users of social networks will adapt over time to the presence of fake news. We therefore investigate to what extent citizens can learn to mitigate the effect of an attacker. For this purpose we train citizens in the presence of a biased signal attacker, a random attacker and an agent takeover attacker for the Barabasi-Alber network. We then evaluate the accuracy of each of these models in the absence of an attack and in the presence of the attack under which they were trained. We summarize our results in Table
Let us first contrast the accuracy of the baseline model (trained without attack) under biased signal attacker with the accuracy of a model trained under this attack scenario. When trained in the presence of an attacker the test accuracy under attack increases from 0.78 to 0.84 relative to the baseline. This can also be seen in Figure [ (B). We conclude that agents can indeed learn to adapt to the presence of fake news. However, this adaptation comes a cost. When trained in the presence of an attacker the test accuracy without attack decreases from 0.92 to 0.88 relative to the baseline. We conjecture that this is because agents learn to trust strong private signals less and are less likely to follow their neighbors' actions. A similar picture can be seen for the random attacker and the agent takeover attacker. To summarize, agents can learn to adapt to the presence of attacker which reduces attack effectiveness. However, adaptation comes at a cost. If trained under attack and tested in the absence of attack, accuracy lower than in the baseline which is trained without attack.
Dynamics of attack and defense: We now take a closer look at the agent takeover attack. Recall that in this scenario, a separate DQN agent is trained to minimize the total reward of the citizens by acting on behalf of a randomly chosen agent. This attacker knows 0 and must then choose an action to convince the citizens of the opposite. Simply choosing the opposite action would be naive strategy that is likely to be detected quickly by the citizens as they learn in the presence of this strategy. Always choosing random action in every period, as the random attacker does, is also
7

---

Under review as a conference paper at ICLR 2020
A
B
Agent type: citizen
Agent type: attacker
Accuracy(t =T)
00.4
AccurssyttTi
W Mm MM
0.8
0.4
0.4
error
0.3
0.3
Accurscyite
0.6
0.2
0.2
0.4
0.1
0.1
0.2
citizen attocker
0.0
0.0
10000
10000
900
1000
Episodes
Episodes
Figure 3: Dynamics of co-evolving attacker and citizen agents on a Barabsi-Albe graph. (A) Red lines are exponentially weighted moving averages. Top: accuracy in the last time step (t T 10) over episodes. Bottom: TD error of Q learning. (B) Close up of accuracy and actions of citizens and attacker early during training averaged over multiple batches. Top: citizen accuracy in the last time step T 10). Bottom: 0).
citizen and attacker accuracy in first time step
quite easily detectable by the citizens. This intuition is confirmed by results reported in Table and Figure
From Table [ we see that the trained attacker clearly "'outperforms" (i.e. more effective attack)
the random action attack benchmark. The bottom panel of Figure (B) shows the accuracy of the attacker in the first time step as a simple summary statistic of its strategy. The accuracy is either 0.5 indicating that the attacker "randomizes" or 0 indicating that the attacker "lies". While Q-learning only allows for deterministic policies the attacker can learn to lie if the attacked agent is in one partition of and be truthful if the attacked agent is in another partition. Since the attacked agent chosen at random, this strategy amounts to randomized attack strategy.
Note further, that neither the attacker's nor the citizens' policy converges as can been seen in the highly variable reward and unstable TD-error in Figure (A) Indeed, panel of Figure 3 (B) the attacker's policy fluctuates between randomization and lying between episodes. These fluctuations can be seen irregular "cycles" in the policy space of the attacker and the citizen. One such cycle in Figure 3 (B) shows the attacker switching to a lying strategy, while the citizens lose "trust" in their private signal (they nearly ignore in the first time step). The result of these two events coming together can lead to substantial, temporary declines in accuracy.
be seen in the bottom
We don't claim that these dynamics are accurate representations of the learning dynamics between users of social networks and pushers of fake news. However, we consider the failure of IQL to converge and the irregular dynamics that emerge more of feature than bug. Indeed these dynamics suggest two interesting, qualitative take aways. First, attackers and citizens are unlikely to converge to stable policies real life. We should expect a continuing cycle of adaptation with periods of a high prevalence of fake news followed by periods of low prevalence. Second, the co-evolution of attackers and citizens can have unexpected effects on the citizens' ability to aggregate information in the social network. As we observe in Figure 3 (B), the presence of neighbors that do not act in accordance with their private signals (i.e. the attacked agents) can lead citizens to learn to mistrust their own private signals.
CONCLUSION AND DISCUSSION
The deliberate manipulation of the public's perception facts via fake news, in particular on social networks, has become a growing concern for policy makers and technology companies alike. De- spite its importance, little is understood about the spread of fake news on social networks. This is in large part due to the technical difficulties involved in studying complex decision making behavior on social networks. We seek to close this gap by developing the (to the best of our knowledge) first framework for the study fake news social networks that is theoretically grounded yet com-
8

---

Under review as a conference paper at ICLR 2020
putationally tractable and flexible. We achieve this by (i) extending standard game of information aggregation to accommodate fake news and (ii) applying state-of-te oh-tth-art deep multi-agent reinforce- ment learning to solve the game.
Our findings suggest a number of interventions that could contribute to making fake news attacks less effective. First, if agents are made aware of the presence of fake news, they learn to adapt and mitigate its effect. However, this adaptation is likely to be accompanied by evolving attack strategies, so any adaptation will not be lead to permanent mitigation. In addition, the adaptation is likely to harm information aggregation in the absence of fake news. Second, keeping private information on social networks private, can make harder for attackers to condition their attacks on network position and general informedness (as the real world analogue of signal strength). This should reduce attack effectiveness. Third, encouraging well balanced social networks can improve information aggregation in general and make fake news attacks less effective in particular.
Our current approach has two main shortcomings. First, does not scale well to larger populations of agents since it requires the entire social network to be trained simultaneously. One way forward is to train agents sub-graphs feeding them an embedding of the graph, see for example Gilmer et al. (017)) and then composing larger graphs at test time. The second shortcoming is the use of IQL in the agent takeover attack scenario. While the lack of convergence leads to interesting qualitative insights. MARL methods that encourage convergence of policies would be desirable here, at least as benchmarks. One possible way to achieve this would be to apply stable opponent shaping to citizen and attacker, see Letcher et al. (2018).
REFERENCES
Noga Alon, Michal Feldman, Ariel D Procaccia, and Moshe Tennenholtz. note on competitive diffusion through social networks. Information Processing Letters, (1066):221-225. 2010.
David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Grae- pel. The mechanics of n-player differentiable games. arXiv preprint 1802::8802000642 2018.
Albert-LászIó Barabási and Réka Albert. Emergence of scaling in random networks. science, 286 (5439):509-512, 1999.
Lesley Chiou and Catherine Tucker. Fake news and advertising social media: A study of the anti-vaccination movement. Technical report, National Bureau Economic Research, 2018.
Kyunghyun Cho. Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.10 1259, 2014.
Morris H. DeGroot. Reaching a consensus. Journal of American Statistical Association, 69:1 69:118- 121, 1974.
Seyed Rasoul Etesami and Tamer Başar. Complexity of equilibrium in diffusion games social networks. In 2014 American Control Conference, pp. 2065-200777 2070. IEEE, 2014.
Jakob Foerster, Yannis M Assael, Nando de Freitas, and Shimon Whiteson. Learning to communi- cate with deep multi-agent reinforcement learning. Advances Neural Information Processing Systems, pp. 2137-2145, 2016.
Jakob Foerster, Richard Chen. Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learing awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 122-130. International Founda- tion for Autonomous Agents and Multiagent Systems, 2018.
Justin Gilmer, Samuel Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning- Learning-Volume 70, pp. 1263-1272. JMLR. org, 2017.
Ben Golub and Evan Sadler. Learning in Social Networks. The Oxford Handbook of the Economics of Networks, 2016.

---

Under review as a conference paper at ICLR 2020
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In Sequential Decision Making for Intelligent Agents Papers from the AAAI 2015 Fall Symposium, 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9:1735- 1780, 1997.
Adam Lerer and Alexander Peysakhovich. Maintaining cooperation in complex social dilemmas using deep reinforcement learning. arXiv preprint arXiv: 2017.
Alistair Letcher, Jakob Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson. Stable opponent shaping in differentiable games. arXiv preprint arXiv: /8/ 1.08469, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529-5333 2015.
Elchanan Mossel, Allan Sly, and Omer Tamuz. Strategic learning and the topology of social net- works. Econometrica, 83(5):1755-1794 2015.
Ardi Tampuu, Tambet Matüisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12(4))e0172339, 2017.
A APPENDIX
A.1 MULTI-AGENT REINFORCEMENT LEARNING
Single-Agent Reinforcement Learning Consider an MDP specified by the following tuple: (S, A, 7)r.7). In the fully observable case the environment has an observed true state, E S, and at each time step the agent chooses action A, which induces probabilistic transition to the next state via the state transition function P(s'|s, a) : S x A x $ > [0. 1]. The reward function assigns an instantaneous reward to each state-action pair, r(s, a) S x A + E. and 7 E [0, 1) is discount factor.
In single agent RL the task of the agent is maximize the discounted reward per episode, Rg Σi=o Yrr- The agent's policy, (aa) x A + [0. 1]. induces state-action caate-action-value function (Q- [ sr, a.). Q-learning aims to estimate the optimal action- function). Q= (F = Ee.41.0 11
value function, Q* a) = max, Q" a). via an estimated Q-function, Q(ssa)).
Training is carried out by collecting samples from the environment in order to obtain Monte- Carlo-Estimates of the expected return. For any sampled transition, the current estimate, Q(s.a), compared to a greedy one-step lookahead using the Bellman optimality operator, TQ(s,a) E, [r + T max,' ((s',a').. We can use samples to evaluate the Bellman update:
Q(s,a)+++ Q(., + (r + 7 max Q(s', ('), Q(x,a)²)
Here h is the iteration number, is the learning rate and > max. Q(s a Q(s, a)) is commonly referred to as the temporal-differeece or TD-error. In the tabular case the Q-values for each state-action pair are maintained separately and the Bellman update is contraction mapping. As a consequence at convergence this iterative process results in the optimal Q- ffncctioon Q(s, a).0 =
Q' (s. a). Finally, Q* a) trivially defines the optimal policy x² (I., a) = (arg max. Q" (z. a), where 8(-) is the Dirac-delta function. In contrast to the tabular case. DQN (Mnih et al. 2015) uses neural network parametrized by large number of weights, Φ to represent the -uuunctoon.
In order to reduce the variance of the update process the average square of the TD-error across a large number of transitions (the batch) used as the DQN-loss. Using backpropagation the parameters of the neural network are updated to minimize the magnitude of the DQN error
(()) 1) PQN
uj + y max Q(z'gs als Φ ), is the target Q( ag: 2] Here v DQN
10

---

Under review as a conference paper at ICLR 2020
function and o is the target network, which contains stale copy of the parameters. This target network helps to stabilize the training.
So far have assumed that the agent has access to the Markov state, of the system. In partially observable settings we augment the MDP with an observation space, Z. and observation function Z is produced by the observation function O(s) S Z.
O(s). In particular the observation We further define an action-observation history 7 E = (Z x UY, which used to condition a
stochastic policy π7 (TaaT) : T x U + [o, 1]. In recurrent deep RL (Hausknecht Stone 2015). this can be achieved using recurrent neural networks, such as LSTM (Hochreiter & SSchmidhuber 1997)
or GRU (Coo al. 2014) which use here.
Independent Q-Learning In multi-agent reinforcement learning, each agent E N receives private observation Oxx, i), where is the agent index and is the observation function. The agents also receive individual reward rf and take actions a; Furthermore the state-transition conditions on the joined action a A A"
In independent Q-learning each agent further estimates Q-function Q'(r', treating the other agents and their policies as part a non-stationary environment. A combination DQN with IQL commonly uses parameter-sharing across agents combined with agents specific index in the ob- servation function in order to accelerate learning while still allowing for specialization of policies.
A.2 IMPLEMENTATION DETAILS
Our code has been submitted. Our neural network architecture implements the IQL. architecture in Foerster et al. (2016).


Reviews:
[{"review_id": "Ske4G4NT9S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors aim to solve the problem of fake news detection in social media. The proposed method is built upon a multi-agent reinforcement learning method. Although the problem of fake news detection in social media has been extensively studied and many method including deep learning have been investigated to help solve the problem, it is relatively novel to use multi-agent reinforcement learning in this field. The proposed method is based on traditional multi-agent deep reinforcement learning approach and the authors extend the conventional framework by introducing the role of attacking agents - agents that can spread biased information or even take over the stance of regular users. The paper has been well written and necessary details for reproducing the experimental results have been provided with a link to the code repository. However, a major concern of mine is the contribution of novelty of the manuscript.\n\nThe main contribution of novelty of the work, as claimed by the authors, is that they come up with a practical solution for fake news detection with deep reinforcement learning. Most research efforts have been focusing on detecting fake and deep-fake content while very few pay attention to utilizing machine learning in learning a best action/practice. This is because the root cause of the widespread of fake news is quite complicated. Besides those who intentionally create and spread fake news and biased content, the innocent users' major problem is how to quickly identify fake news from massive amount of information flows. The idea that building a practical fake news prevention solution by using multi-agent reinforcement learning seems to have underestimated the complexity of the misinformation challenge. For example, three high-level suggestions/solutions proposed in the manuscript include social network users should be more aware of the presence of fake news, keeping private information private on social networks, and encouraging well balanced social network structures. My question is that how we can apply these solutions in the real world? Therefore, I think this piece of work is more theoretical rather than practical.\n\nIn terms of the technical part, the authors propose to introduce agents for fake news and biased information. The technical solution is solely based on multi-agent reinforcement learning and the extension is straightforward. The assumption that there is only one kind of role for fake news dissemination in social networks again underestimates the complexity of the problem in the real world. E.g., there are users who intentionally create fake news in social networks, and also users who are not aware of a piece of information being fake and yet still deeply believe what they spread is true. I would suggest the authors to find more related work in the field of information diffusion, where researchers have long been focusing on competitive information propagation in social networks with multiple parties (such as political campaign and word-of-mouth social marketing). \n\nIn conclusion, I think the work is well-written and quite interesting - solving an emerging and important problem from a new perspective. However, both the hypothesis and the technical solution are lacking enough contributions of novelty. I would like to suggest the authors either make the solution actually practical or focus more on the theoretical part of competitive information diffusion in social networks.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HJlH_4g-cr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a model under which to study social networks under attacks attempting to propagate misinformation. It proposes a theoretical model based on assumptions on what kinds of graphs are common in social media and what kinds of attacks take place. While this could be interesting, the work presented falls short of what it promises, i.e. to develop a practical model of fake news on social networks,  because many of the assumptions made about the phenomena under study are unrealistic. In more detail\n\n- There is a lot of research looking at social network graphs and analyzing them. As an example, here is a paper by Kate Starbird: https://www.aaai.org/ocs/index.php/ICWSM/ICWSM18/paper/view/17836/17028 \nI believe that there is little reason to generate data if one can collect them\n- Even if generating the data can be justified, the graphs on which the methods are studies are 10-12 nodes. I can't see why such a low number was chosen given that the data is simulated, but it doesn't allow to assess whether the methods proposed would be practical in a real social network\n- There is work suggesting that misinformation spreads faster than information: https://science.sciencemag.org/content/359/6380/1146\nThus it would make sense to take this into account in designing the graph theoretical model. Given this though, it is unlikely that the assumption that the social network will converge to the truth.\n- Assuming that the social network graph remains fixed over time is also unrealistic. One can study how Twitter networks evolve over time\n- The modelling of the users as merely voting on the truth or false value of a claim is not what happens in most social networks in which users fave/like, share, etc. Furthermore, it doesn't make that users want to prevent other from copying them. Being retweeted is a sign of influence, and users want to be influential.\n- The attacks described do not seem to be grounded in any evidence/research on how misinformation propagates and what attackers actually do, so I can't accept the results of the analyses that use them.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "ByxSJJLjYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: I thank the authors for their response and for improving the paper. However, I maintain my position that the paper lacks a significant technical contribution to learning algorithms and that the applicability of the proposed approach is remains questionable in the current state.  \n\nSummary:\nThe paper proposes the  use of deep multi agent reinforcement learning (DMARL) for modelling fake news propagation and detection in social networks. The agents observe an informative yet noisy private signal and the actions of their neighbors (in the social network graph) and have to guess whether a claim (related to the received signal) is true or false. The fake news is modelled as an adversarial attack to the graph that either provides a hand-coded biased private signal to one of the agents or replaces one of  the agents with an RL policy trained to minimize the total reward of the agents in the graph (i.e. social network).\n\nMain Comments:\nI lean towards rejecting this paper because I do not find the methodological contribution to be significant enough to be published at ICLR, given that the main contribution is applying current techniques to a novel toy domain. While this paper attempts to apply DMARL to a new domain with real-world relevance, the authors only consider a toy example and make strong assumptions that are likely to break in the real world. Hence, it is not at all clear whether or how the conclusions of this paper would translate to more realistic scenarios of fake news in social networks. While strong assumptions and toy examples are reasonable for showing algorithmic improvements, this paper does not propose any improvement to core DMARL algorithms, but merely applies current methods to a new toy domain. \n\nI am also concerned about the lack of comparison with other approaches to information aggregation in social networks. While I admit I am not familiar with that literature, I would still find it useful to provide some comparisons with non-DMARL (e.g. heuristic or game theoretic) approaches or at least some motivation for not comparing against those methods. The authors qualitatively describe those methods and their shortcomings, but the experimental section does not support those claims due to the lack of comparison. Despite all these concerns, the paper does indeed open-up numerous research directions and I can imagine follow-up papers being written that relax some of the current assumptions. \n\nOther Questions / Comments:\n\n1. Can you provide  some motivation for choosing to model the social network as a Barabsi Albert graph and why this is a reasonable modelling choice?\n\n2. What happens if instead of a clustered or balanced graph, you have some combination of the two? It seems to me like that would be a more realistic scenario (i.e. a large graph containing subgraphs with different structures). Can the framework generalize to that? How would the conclusions change?\n\n3. Is there any evidence that the conclusions supported by the experiments in this paper hold in real-world social networks and model some realistic aspects of social network dynamics?  Without such evidence, it is difficult to assess the relevance of this work for the  real-world application. Since the behavior of the agents in the graph is not guaranteed to be optimal / a best-response or even stable, is it at least a good approximation to human behavior in social networks? It would also be useful to show more comparisons against best-response or heuristic agents.\n\n4. What is  the motivation behind considering a fixed budget of bias? Why not instead have a fixed number of agents that you will be biased? I think it would be informative to compare against applying beta = 3 to two citizens,  along with the focused and spread scenarios.\nThe legend in Figure 2 A & B is slightly confusing. I’d suggest using different styles for “private signal optimal” and “all signals optimal”. \n\n5. Can you include error bars in Figure 2 D?\n\n6. Can you provide results with a heuristic attacker that always lies about the claim? I read your intuition of why  you believe this wouldn’t be stronger than an attacker that is trained with RL together with the other agents, but is it actually true in practice, do the agents really learn to easily detect the “lying” attacker and distrust it? Based on your results, doesn’t it mean that one can find a heuristic attacker that has an equivalent behavior to the learned one ? Can one build even stronger hand-tuner attackers  based on heuristics?\n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper aims to model fake news by drawing tools from multi-agent reinforcement learning. After the discussion period, there is a consensus among the reviewers that the paper lacks novel technical contributions. The reviewers also acknowledge that paper also doesn't quite deliver a practical solution as claimed by the authors.
Decision: Reject