Distributed News Monitor System
By: D Xiang
Tags: Machine Learning, Mongodb, Streaming, Distributed Systems, Kafka

(4) Data modeling:

Based on our initial goal on classifying news from content, we apply a bi-directional RNN with attention mechanism to the news. This mechanism leads the model to give different weights to individual words and further, different sentences.

(4.1) Hierarchical Attention Networks[5]

Hierarchical attention network along with its supported parts: sequence encoder and hierarchical attention should be introduced. The sequence encoder is a stack of GRU cells that each propagates an input within a sequence and pass it into the next hidden states with updated and reseted controls. Hierarchical attention structure can be divided into four parts: a word sequence encoder, a word-level attention layer, a sentence encoder and a sentence-level encoder. In terms of word sequence encoder and sentence encoder, single words are embedded into vectors and bi-directional GRU takes the word embedding from both forward and backward directions of the sentences to get this two hidden states. By concatenating the hidden states for one word, we are able to get the hidden annotation of a specific word with the summarization of nearby words. The sentence encoder works in the way. The forward and backward bi-directional GRU take in the sentence embedded vector respectively and concatenate to get the hidden annotation emphasised on the current sentence. With regard to the word-level and sentence-level attention, a context vector is randomly chose and used to be compared with when finding attention weights. The hidden annotations from the previous step are taken into a multilayer perceptron and we compare this obtained hidden representations with the previous mentioned context vector.

(4.2) Model fitting and training process

562 fake and real news are randomly separated to training data set(448 news) with test data set(112 news) for 5 times(i.e: 5 training set with 5 test set). Top 16000 words are selected and ranked by its own tf-idf. The maximum number of sentences for a news is fixed to be 100 and we only retain the words appearing within these 16000 words. We then load the pre-trained word2vec model to initial the word embedding matrix. For hyperparameters, we set the embedding dimension to be 300 and the GRU hidden dimension to be 50. In the training process, we use the batch size of 50 and use the gradient descent to train the model.

Stage II:

(1) Data source crawling: News API is adopted to crawl daily news headlines as the preliminary step. Then, the power of Name Entity Recognition will aid us in extracting the most popular daily news keywords which will be triggered once a day. According to these keywords, we are off to the hunt for the Twitters that are related to keywords. However, there are still too many Twitters to crawl, some of them might have a couple likes whereas the other one might has more than a thousand reactions, so standards and criteria must be set. Considering the feasibility and scalability, only the Twitter that receives more than a thousand reactions(i.e. number of comments and likes combined) will be crawled and analyzed. Then, we use Twitter Search API to scrape news URL, Twitter URL, comments, and etc. for further cleaning, preprocessing, and analyzing.

(2) Data Preprocessing: We save crawled data as dataframe for convenience of aggregated computation. From the crawled columns, the comments analysis part would receive these following columns for further analysis: Twitter content, comments content, news keyword, news URL, Twitter URL, retweet count, and likes count. It would trigger the distributed web crawler every minute to scrape the real-time comments off the Twitter that we are interested in. From on the crawled comment content, each comment is tokenized into single words using NLTK, dropping stopwords at the same time of finishing morphology, to prepare for further analysis and visualization.

(3) Data Analysis: For data analysis, the NLTK sentiment analyzer is utilized while taking in the tokenized comments and outputting the polarity sentiment score for each comment. A common consensus is that the more likes a comment receives the more likely the public is to agree with this comment. Therefore, a weighted average feature is introduced to the sentiment analysis such that the sum of the total number of likes across all the comments plus the number of comments is calculated first. Then each comment receives a weight that is equal to its number of likes plus one(one is added to symbolize the viewpoint of the person who posts it) divided by the sum from above as shown in equation(1). After calculating this weight for each individual comment, it is multiplied with each respective aggregated polarity sentiment score. Finally, these weighted scores will be summed to arrive at the final result that is between [-1, 1] to represent the general attitude of the public toward a matter. The closer it is to 1, the more positive the reaction is, whereas the closer it is to -1, the more negative the responses are, and 0 stands for a more neutral outlook. This is constantly performed on the fly, utilizing the Spark streaming function to achieve a real-time output.

Figure 6| Formula for calculating comments weighted sentiment scores

(4) Data Visualization: All the columns that are crawled and calculated are all stored in MongoDB for better visualization speed. On the web page the following items will be shown for each individual news: a calculated weighted sentiment score, top ten comments with the most likes, a word cloud, a line chart that shows the number of comments against time for a more clear visualization. As for the word cloud, POS-tagging is used to only count the adjectives and thus displaying them. The scatters of all polarity sentiment scores for each news is also plotted to visualize the attitudes from all readers to the original tweet.

Stage III:

System Design and Implementation: Since our goal is to build a well-rounded pipeline from start to finish with considerable efficiency, feasibility,and scalability. Spark streaming function is implemented between the components so that the result can be as real time as possible. Kafka is used when triggering the news classifier which it happens once a day, as well as when triggering the distributive web crawler every minute for the comments analysis. After these components have finished the calculation and determination from the model, the result would be passed through the Kafka streaming into MongoDB for our frontend web. Every news has its own unique identifier, the analysis from these two components would merge into one single document.

Moreover, we choose to implement a distributed web crawler because of scalability and the possibility to exploit more options for further investigations later on. Therefore, a separate distributed web crawler for each component written in PySpark is deployed in the pipeline.

Techniques and Technologies List

Web Scraping: news API, Python newspaper, Twitter Search API, Selenium.

Data preprocessing: NLP(NLTK, corpus, SentimentIntensityAnalyzer, SpaCy, en_core_web_sm), Pandas, pyspark ML Pipeline, pyspark SQL.

Data modeling, analysis: Keras, Tensorflow, gensim, numpy, scipy, sklearn.

Data visualization and Web UI: seaborn, matplotlib, Node JS, React, RESTful API.

Data storage and stream-process: MongoDB, Apache Kafka.

Evaluationused Tensorflow and

(1)Model

In addition to the bi-directional RNN model described above, we applied four additional model which are logistic regression, multinomial Naive Bayes, random forest and XGB. Accuracy, recall and f1 score are used as the criteria to compare the performance of these classifiers. The bi-directional RNN model has better performance over the other classifiers under all these three statistics.

used Tensorflow andused Tensorflow andFigure 7| Performance comparison between different classifiers