Real-time COVID19 Public Sentiments Using Tweets
Tags: Sentiment Analysis, Real Time Analytics, AI, Data Science, Streaming Architecture

During the COVID19 pandemic, we also want to contribute as data scientists but in different perspectives. We have seen many dashboards reporting the numbers of the confirmed and tested cases. However, we are more interested in understanding how the public react to the pandemic. We then designed and quickly implemented a pipeline that streams in the live tweets, then feeds into the data pipeline, including: i) filter the tweets; ii) predict the sentiment of the tweet content using deep-learning-based NLP algorithm; iii) identify the location of tweet’s author; iv) visualize on the dashboard.

The dashboard can be seen from http://covid19.pata.ai

Data sources: tweet streaming API with keywords. We implemented consumers to subscribe streaming tweets, with particulary keywords, e.g. covid19, pandemic, coronavirus and those related to the current affair. Data injection: Logstash from the ELK stack (or currently called Elastics Stack) as the injection tool. A third-party filter plugin (https://github.com/lucashenning/logstash-filter-rest) was used to call our sentiment model (behind an async http server), and append the sentiment information to the streaming message. Sentiment Model: We quickly implemented a simple RNN-based sentiment model with fasttext as the tool to embed words. The output is logists of 3 followed by softmax layer, representing the probablity of the tweet to be postive, neutral and negative. Model is not in the optimal states. We will finetune the model during further developments. Message queue: A kafka instance running on k8s is used as the message queue. the Logstash output plugin keeps producing message to the queue. Geocoding: we need to geocode each author’s location from a string to normalized format (latitude, longitude and the official country code and admin names). Currently, we use a pool of the geocoding services, including Here, GoogleV3 and etc. However, the pricing is very expensive in terms of API calls. Therefore, we are implementing our own geo services using Pelias (https://github.com/pelias , thanks for the contributions from their great works). The geoservice is behind an async http server with restful API. Caching: we implemented cache in multiple components to reduce external API calls, reduce lantecy and improve overall throughputs. We use redis streams to cache the large time series data and a dedicated http server is running in front of the redis as the API endpoints. Dashboard: Grafana is used to build the dashboard with a third-party worldmap plugin (https://grafana.com/grafana/plugins/grafana-worldmap-panel). The grafana can use Json as data source which connect to the Restful API of the redis server.

As shown in the figure above, we have only put two panel at this stage, the world map and a table. The dots on the worldmap are live tweets streamming in from Twitter API, and classified as positive(green), neutral(blue) and negative(red). The tweet contents are also embeded on the map, which you can access that by hovering the mouse on the dots. A 5-min time window is maintained, which means only the latest tweets within 5 minutes are displayed.

The right panel is a table has the real-time summary information of the country-wise sentiments. It includes total number of real-time tweets within in 24 hours, and the ratios of positive, neutral and negative sentiments. All the numbers are updated in real-time. But in practive, we limit the refresh rate to 10 seconds, in order to reduce the presure to our backends. The numbers will be reset every 24 hours. N.B. the numbers are not ‘real’ reflects of the public sentiment. The results are affected by the tweets sampled, and the seniment model accuracy. However, the numbers can be recogized as sampling from the ‘true’ public sentiment distribution.

We develop this tool to understand how public reacts to the pandemic, and seeing the world fighting against the virus. We don’t want to see the panicness from the public, but we really want to provide a tool for people to understand how the world reacts during this dark peroid emotionally.

There are other hightly recogized dashboards, such as

John Hopkins COVID19 dashboard (https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6) Dashboard from ravenpack.com (https://coronavirus.ravenpack.com/)

We sincerely thank the open source projects and the coummnities:

Pytorch Logstash Kafka Geocoder Redis grafana fasttext k8s helm

Stay safe, and we will go through this together.