DEIM Forum 2019 B1-5

GAN によるフェイクニュースの分類と生成
Can we learn more about fake news by generating it directly
Miao XIAO† , Rakesh AGRAWAL†† , Kenki NAKAMURA† , Tianwei CHEN† , and Qiang MA†† School of Informatics, Kyoto University
Yoshida-honmachi, Sakyo-ku, Kyoto, 606–8232, Japan†† Data Insights Laboratories, University of Wisconsin-Madison
1210 W.Dayton St. Madison, WI 53706–1613, USA
E-mail: †{mxiao,nakamura-kenki,chentianwei}@db.soc.i.kyoto-u.ac.jp, ††rakesha.prof@gmail.com,†††qiang@i.kyoto-u.ac.jp
Abstract Nowadays, more and more people tend to consume news from social media. However, fake news has
been easily and widely propagated by social media due to there is no significant third-party fact-checking the contents of news. It is a challenge for us to classify fake news articles into diﬀerent types and notify users reliabilityof news. Related work primarily utilizes machine learning methods to do classification which is based on a large
amount of fake news data. However, since it is so diﬃcult to recognize fake news among all the news that wecannot collect fake news easily by some heuristic methods like web crawler, in which case vast manual eﬀorts are
required. On the other hand, news patterns are always constantly updated with new events. If we blindly extract
fixed features from an existing dataset, our model will be over-fitting. Our proposed method tries to solve these
problems by training a pair of adversarial neuron networks simultaneously: Classifier judges whether input fakenews data is from Generator or not and categorize real-fake-news, then leak information to guide Generator to write
fake-fake-news. Generator captures the distribution of real-fake-news dataset and fools the Classifier. We call thismethod as Fake News GAN which is short for fake news generative adversarial networks.
Key words Fake News, generative adeversarial networks, deep learning, natural language processing.

1 Introduction

tional media technologies: there are no third-parties to filter,check the fact, or to add editorial judgments to these con-

Nowadays, social media plays a more and more important

tents relayed by social media. Furthermore, sometimes, arole in human s daily life. We almost use social platforms

user without reputation can reach as many readers as NHK,every day such as Twitter, Line or otherwise to check theor the New York Times. Once he/she relays fake news amonglatest information or just send messages to friends. Social

his/her followers, false information will be propagated by so-

platforms do really provide convenient and seamless accesscial media rapidly and widely.

to information.

Fake news is news, stories or hoaxes created to deliberately

As a result, more and more people tend to consume news

misinform or deceive readers（注 1）. Fake news has raised morefrom social media: about 62 percent of Americans get news

and more concerns since the 2016 US presidential election.

from social media [1] and more than one-third people from

The most discussed fake news stories tended to favor Don-the UK and US will get news from social media firstly when

ald Trump over Hillary Clinton, Such as Hillary indictmentusing a smartphone [2]. The reason is that it is often moreimminent（注 2）. So a number of commentators have suggested

timely and less expensive to comsume news by social media

that Donald Trump would not have been elected president

compared with traditional news organizations, such as news-

without fake news.

papers or televisions; and it is more convinient to share, add
comments on, or discuss the news with friends directly.
Aside from all the merits, on the other hand, there are

Fake news can aﬀect our society proufoundly. Two typicalways that aimed to interdicting the propagation of fake news
（注 1）：https://www.webwise.ie/teachers/what-is-fake-news/

also some fatal drawbacks due to the structure of social me-（注 2）：https://www.politifact.com/punditfact/statements/2017/oct/31

dia which is dramatically diﬀerent from the previous tradi-/freedomjunkshuncom/fake-news-site-claims-hillary-clinton-flies-ukrainhas been verified to be eﬀective: fake news identification andout from social media rather than waiting for the score byclassification.

third-party fact-checking.

Identify fake news directly requires a large number of man-

2 2 Tensor Decomposition in Ensembles

ual eﬀorts, furthermore, this method cannot remind the pub-

Hosseinimotlagh et al. [8] proposed a method by modelinglic of the reliability of news from social media timely. So thethe fake news corpus as a 3-mode (article, term, term) tensor

major challenge for us is to construct a model which is able

which captures spatial terms relation and article-term rela-

to classify the fake news articles into diﬀerent types and re-tions, and then use CP/PARAFAC decomposition to identify

mind readers timely. On the other hand, for deep learning

latent groups of articles. The highlight of this paper is thatbased text-classify models，we need a large amount of train-

they introduced an ensemble method which leverages multi-

ing data, but fake news dataset is rare and it is too diﬃcult

ple decompositions of the tensor to further refine the latent

to identify fake news from the real without manual eﬀorts.

groups and produce a clean categorization of articles, which

And the most important feature of fake news is that its’ pat-attends higher accuracy than the state of the art. The in-terns change rapidly, which means that as time goes on，thetuition behind the method is that news articles that tend

way to write fake news articles can be changed frequently.

to frequently appear surrounded each other among diﬀerent

So, concentrating too much on the current fake news will

rank configurations of tensor decomposition.

make our model be overfitting.
Our main contributions are as follows:•

2 3 SeqGAN
GAN [9] is short for Generate Adversarial Network andWe proposed a LSTM based neuron network, which

originally designed for generating images, due to the stochas-

converts the words of news articles into dense vectors thentic gradient descent algorithm [16] and consecutiveness ofutilizes LSTM [3] to classify the fake news articles. The whole

pixels.

network is trained and tested by the real-world fake news

word vectors with on consecutiveness, so we must leveragedataset from Kaggle（注 3）. The experimental results verified

the power of reinforcement learning rather than BP gradientthat our method achieves better performance compared todescent to guide the generation process.

the recent studies.
•

But in NLP tasks, the training data are usually

There are several related works for GAN in NLP:We proposed a generative adversarial network based

MaskGAN [19], SeqGAN [10], and LeakGAN [11] which haveon the preceding LSTM neuron network, we called Fake News

been verified to perform very well in generating semantic

GAN, can generate and classify fake news simultaneously

sentences. SeqGAN utilizes Policy Gradient algorithm [17]which avoids over-fitting and covers the randomness of fakeas the optimization method. The general settings are: re-news patterns eﬀectively.

gard generator as an agent, word token generated by it at

2 Related Workevery time stamp as action, and all the tokens generated before as state and then use a reward function formed by the

Fake news raises major concerns since the 2016 US elec-

real possibilities of complete sentences judged from discrimi-

tion. There exist several related work to identify or classify

nator. For incomplete sentences generated at a certain time,

fake news articles recently. In this part, we will introduce

Monte Carlo Tree Search [11] is used for completing them.

several related methods and some deep learning technologies
used in our proposed methods.
2 1 Online Fact-checking

3 Deep Learning Based Fake News Classification

Shao et al. [7] proposed an online fact-checking platform for

From above related work, we find it more eﬃcient to focus

fake news named Hoaxy. This application collects credibleon the fake news text itself. By classifying fake news articles

scores of news articles from several online fact-checking web-

into diﬀerent types, we can remind readers of the reliabilitysites, such as Snope.com, PolitiFact.com, and FactCheck.org.

of news timely. In this section, we proposed two progressive

This platform does really work well due to the eﬃcient scores

deep learning based methods which classify fake news withof these websites. On the other hand, information can behigher accuracy than the state of the art.

propagated by social media rapidly, both evaluating the fake3 1 LSTM Based Method

news and collecting the scores spend too much time to avoid

In this part, We construct multiple neuron networks tobreaking fake news to confuse people. For example, Barackclassify fake news text. As shown in the figure1, we get fakeObama being injured by an explosion at the White House.

news text as the input of word embedding layer and thenThus, we should identify fake news as soon as they comeuse 3 kinds of word embedding methods(one-hot, GloVe, and（注 3）：Megan Risdal. Fake news dataset. https://www.kaggle.com

FastText) for comparisons. After that, input the whole or the/mrisdal/fake-news

part of text matrices formed by words vectors to the LSTM.

Finally，a softmax layer is utilized to classify the feature vectors from LTSM. The following parts will introduce the wordembedding and LSTM layer in detail.
3 1. 1 Word Embedding
Recently, distributed representation of words has been
widely utilized as the pre-training of natural language processing tasks. Unlike the tensor decomposition, news text isrepresented as a words co-occurrence matrix.
In this model, we use two typical word embedding methods, GloVe [12] and FastText [13], and one-hot as the comparison, to pre-train our neuron networks. GloVe is an unsupervised learning algorithm. Unlike the Word2Vec [14] model,
the training process of GloVe is performed on aggregatedglobal word-word co-occurrence statistics from a corpus, andthe results indicate the linear substructures of the word vector space. The following formulation indicates the loss function of GloVe model:
J=

V∑

Figure 2 Fake News GAN

RNN, hidden state at the previous moment is not completely
washed away, thus this kind of structure enhances its ability to process long text sequences and solves the vanishinggradient problem.
3 2 Fake News GAN
Based on the results of the experiments of the preceding
LSTM based neuron network, we find it more eﬃcient tof (Xij )(wiT w̃j + bi + b̃j − logXij )2

(1)

i,j=1

classify the fake news articles by deep learning than our baseline tensor decomposition. On the other hand, deep learningwhere wiT w̃j are the word vectors we want to get, bi and b̃jbased methods require plenty of training data to modify aare bias variables, logXij is the weight function and Xij islarge number of parameters. There are only rare datasetsthe entry from word co-occurrence matrix.

existing since it is so diﬃcult to recognize fake news from

FastText is based on Word2Vec model but every single

the all the news that we cannot collect fake news easily byword is not the smallest part within the model. It assumes

some heuristic methods like web crawler, in which case vastthat n-grams of characters form a single word, and n could

manual eﬀorts are required.

range from 1 to the length of the word. For instance, learn
is composed of [lea, lear, learn] etc.

Since we use deep learning, we must pay attention to avoid
over-fitting. The most worthy information of one news arti-

3 1. 2 LSTM Layercle describes events from who, what, when, where, how, andLSTM is short for Long Short-Term Memory Network andwhy. And if we change any part among them, we can see it

widely used in some NLP tasks, such as Machine Translation,

as fake news. So due to the special patterns and randomness

Text Classification, QA and other fields. LSTM is based onof news articles, even if we collect all the current fake news as

RNN but adds memory cells and three control gates to eﬀec-

training data, it will be changed quickly in the near future.

tively control historical information especially in long text

If we blindly extract fixed features from a current existingtasks. Input gate controls the extent of information fromdataset, our model will be over-fitting.

the input vector at current moment; forget gate controls how

In this part, as shown in figure 2, we proposed a GAN

much the historical information influence the information at

based neuron network to improve the accuracy, avoid over-the current moment; output gate controls the extent to which

fitting, and get more training data by generating fake-fake-the value in the cell is used to compute the output. Unlike

news directly due to the randomness of the generator.
3 2. 1 Classifier
GAN trais two neural networks Generator and Discriminator simultaneously. General discriminator tries to judge
whether the input image data is from the generator or not,and generator tries to capture the distribution of real dataset
then fools discriminator mistakes its inputs as real. The
whole process can be formalized as a min-max game withthe following loss function between G and D:
max min V (G, D) = EX 〜pd (x) logD(X)+EZ 〜pz (z) log[1−D(G(z))]
G

D

(2)

where pd (x) is the data distribution from training data,pz (z)Figure 1 LSTM Based Network

is a prior on input noise variables. In our network, discrim-inator will be modified as a classifier which categorizes fakenews articles into several types and judge the real or fakeat the same time. Since words vectors of news articles has

Table 1 Fake News Labels from BS Detector
Category Names
Biasno consecutiveness, we must leverage the power of reinforcement learning at every turn of the training process: the classifier will leak the extracted features information from theConspiracy
Satirereal-fake-news as a reward signal to guide the generator toimitate the real.

Sources that are well-known promoters of
kooky conspiracy theories.
Sources that provide humorous commentaryon current events in the form of fake news.
misogyny, homophobia, and other formsof discrimination.
Sources that promote pseudoscience,

Generator tries to capture the distribution of the real-fakeJunksci

metaphysics, naturalistic fallacies, andother scientifically dubious claims.

by classifier. In our network, generator also receives the reward signal from classifier to guide the generation process.

and gross distortions of fact.

Sources that actively promote racism,Hate

3 2. 2 Generator
news dataset to make generated sentences be judged as true

Descriptions
Sources that traﬃc in political propaganda

State

Sources in repressive states operating
under government sanction.

But due to the instability of existing reinforcement learning applied to GAN, as the length increases, the generated
sentence will be more and more non-semantic. In leakGAN
and seqGAN, the limitation length of generated sentences isabout 30 words, but fake news articles has more than 300
words in general.
Our idea is to mine the abstract of the whole text before inputing them into the network, for example, by using
textrank [20], which means that classifier categorize news articles by only using abstracts. This idea has been verified
to be eﬀective from LSTM based neuron network. And thegenerator also only need to generate the fake news abstract
which contains less than 30 words.

4 Experiments

naive one-hot method is that we want to prove the qualityof word vectors impact the results a lot.
Furthermore, based on word vectors, in order to make ourneuron network understand the text in more detail, we use
Stanford Parser [15] to mine the sentence structures in grammar, for instance, find the subjects and objects within one
sentence.
From figure 3 and figure 4, as expected, we can find that
GloVe and Fasttext performed much better than one-hot,but grammar structures impact little on results. Moreover,in satire and junksci, we find apparent downtrend as thenumber of top news increase. We think this is because themodels are overfitted or the instability of BS Detectors.
4 3 CNN based network

In this part, we conduct a series of experiments on our proposed LSTM based network by using various forms of word
vectors as the input of the LSTM network. As for experiments on GAN in this part will be held on classifier andgenerator respectively.
4 1 Dataset
We evaluate the accuracy of all the proposed networks
on the same real-world dataset as Tensor Decomposition [8]from Kaggle for comparison. This dataset contains more

CNN based network will be utilized as the classifier of our
Fake news GAN, and we think it can extract features of text
easily than LSTM with lower computational cost. In thisexperiment, we used the text-rank algorithm to extract theabstract of every piece of fake news corpus then delete items
longer than 30, for which we can ensure this pre-processed
dataset can also be applied to the generator. The CNN network is quite light and only use the one-hot word embedding
method with several convolutional and pooling layers. We

than 12,000 fake news articles which are labeled by BS detectors（注 4）. Table 1 demonstrates various types with the description of fake news.
4 2 LSTM based network
We only use the news text and labels of the dataset fortraining and validation. After deleting the stop words, as
we have mentioned above, 3 kinds of word embedding methods will be performed for experiments: one-hot word vectors
in 1,000 dimensions, GloVe in 300 dimensions, and Fasttext
in 300 dimensions. Both GloVe and Fasttext are advanced
word embedding methods, and the reason why we still use（注 4）：B.S.Detector. Fake news classifications. http://bsdetector.tech,

2017.

Figure 3 Results from top 30 news of bias, conspiracy, and satire
respectively.

Table 2 Fake Fake-news Examples
a few people had a finding of a trump supporters to the presidential
1 election and promises his connection to be the fact of the servers
on the accused of dating.
Hillary Clinton s campaign chair massing the cities and for emails
2 nor will stanss work in a collect her child from action here to geta supporters who support Hillary Clinton s campaign chairman.
Investigation of the remaining millions of the report issued by the3

white house with republicans in the united states is said sensationof the republicans would stand for healthy ones on Wednesday
through the life.
The video described in the republicans is a fact to he stop near

Figure 4 Results from top 30 news of bias, junksci and state respectively.

4 georgial community and promised to the will be out that they are
the treated is a regular.
The emails start as hillary clinton email scandal that he is not

apply 10,195 pieces of news abstract as input and the wholenetwork are trained for 3 epochs in 1950 steps. Finally, itachieved a quite well result as figure 5 illustrates.

5

enforcing the law some it s nom that the role the donald trump
and healthy ones of the fbi and investigation into the presidential
election of his committed to grab his pack and industry.

4 4 Fake Fake-News Generating
Here, we modified the seqGAN model to be fit for our

GAN for each kind of fake-news. And at present, our workdataset and purpose then get some generations seems like tois just concentrated on how to classify the fake-news due tobe semantic. And here are some examples of our results inthe dataset, in the future we may find the datasets labeledtable 2.

by fake and real to train the Neuron networks to identify the

5 Conclusion and Future WorkIn this paper, we proposed two neuron networks for fakenews classification. LSTM based network has been verified
to be more eﬃcient than the baseline Tensor Decomposition

fake news. And recently, some deeply contextualized word
representation methods, such as BERT [21], have been verified to improve the accuracy of basic NLP tasks a lot. We
want to apply them to our models in the future.

6 Acknowledgements

by our experiments. As for GAN based network, we just construct the generator and classifier respectively, and from theresults, we can see that CNN based classifier achieves better

This work is partly supported by JSPS KAKENHI(16K12532)
and The Kyoto University Foundation.

accuracy than LSTM. But on the other hand, the results ofthe generator is not well as expected.
In the future, we need to combine our classifier and generator together to get a better result and still need to conduct
more experiments to prove our idea. Besides, in order tolabel the fake fake-news, we consider setting one particular

Figure 5 CNN based news abstract classification result

References[1] Allcott, Hunt, and Matthew Gentzkow. ”Social media andfake news in the 2016 election.” Journal of Economic Perspectives 31.2 (2017): 211-36.
[2] Gottfried, Jeﬀrey, and Elisa Shearer. News Use Across Social Medial Platforms 2016. Pew Research Center, 2016.
[3] Hochreiter, Sepp, and Jürgen Schmidhuber. ”Long shortterm memory.” Neural computation 9.8 (1997): 1735-1780.
[4] Zafarani, Reza, Mohammad Ali Abbasi, and Huan Liu. Social media mining: an introduction. Cambridge University
Press, 2014.
[5] Centola, Damon. ”The spread of behavior in an online social
network experiment.” science 329.5996 (2010): 1194-1197.
[6] Kempe, David, Jon Kleinberg, and Éva Tardos. ”Maximizing the spread of influence through a social network.” Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2003.
[7] Shao, Chengcheng, et al. ”Hoaxy: A platform for tracking online misinformation.” Proceedings of the 25th international conference companion on world wide web. International World Wide Web Conferences Steering Committee,
2016.
[8] Hosseinimotlagh, Seyedmehdi, and Evangelos E. Papalexakis. ”Unsupervised Content-Based Identification of Fake
News Articles with Tensor Decomposition Ensembles.”(2018).
[9] Goodfellow, Ian, et al. ”Generative adversarial nets.” Ad-vances in neural information processing systems. 2014.
[10] Yu, Lantao, et al. ”SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient.” AAAI. 2017.
[11] Guo, Jiaxian, et al. ”Long text generation via adversarial training with leaked information.” arXiv preprint
arXiv:1709.08624 (2017).
[12] Pennington, Jeﬀrey, Richard Socher, and Christopher Manning. ”Glove: Global vectors for word representation.” Proceedings of the 2014 conference on empirical methods innatural language processing (EMNLP). 2014.
[13] Bojanowski, Piotr, et al. ”Enriching word vectors with subword information.” Transactions of the Association for Computational Linguistics 5 (2017): 135-146.
[14] Mikolov, Tomas, et al. ”Distributed representations of words
and phrases and their compositionality.” Advances in neural
information processing systems. 2013.
[15] Danqi Chen and Christopher D Manning. 2014. A Fast andAccurate Dependency Parser using Neural Networks. Proceedings of EMNLP 2014.
[16] Robbins, Herbert, and Sutton Monro. ”A stochastic approximation method.” Herbert Robbins Selected Papers.
Springer, New York, NY, 1985. 102-109.
[17] Sutton, Richard S., et al. ”Policy gradient methods for reinforcement learning with function approximation.” Advancesin neural information processing systems. 2000.
[18] Browne, Cameron B., et al. ”A survey of monte carlo tree
search methods.” IEEE Transactions on Computational Intelligence and AI in games 4.1 (2012): 1-43.
[19] Fedus, William, Ian Goodfellow, and Andrew M. Dai.
”Maskgan: Better text generation via filling in the .” arXiv
preprint arXiv:1801.07736 (2018).
[20] Mihalcea, Rada, and Paul Tarau. ”Textrank: Bringing order
into text.” Proceedings of the 2004 conference on empiricalmethods in natural language processing. 2004.
[21] Devlin, Jacob, et al. ”Bert: Pre-training of deep bidirectional transformers for language understanding.” arXiv
preprint arXiv:1810.04805 (2018).