Finding the urgent blood/plasma requirements on Twitter using Spark Streaming
By: Annapoorani K
Tags: Spark, Spark Streaming, Scala, Twitter

Introduction:

In this code, I have used spark streaming to

1) identify the tweets related blood or plasma requirements

2) filter out the tweets such that only the tweets related to urgent or immediate blood or plasma requirements are identified

3) Extract the date, userID, location, and hashtags used for such tweets and store it in a table and a parquet file

Technologies Used:

1) Spark version 3.0.1

2) Scala version 2.12.10

3) Java 1.8.0_271

Initial Configuration:

1. First, we import the names of the Spark Streaming classes and some implicit conversions from StreamingContext into our environment

import org.apache.spark._

import org.apache.spark.SparkContext._

import org.apache.spark.sql.functions.{col, lit, when}

import org.apache.spark.streaming._

import org.apache.spark.streaming.twitter._

import org.apache.spark.streaming.StreamingContext._

import org.apache.spark.sql.SparkSession

import org.apache.spark.sql.SaveMode

import org.apache.spark.rdd.RDD

import Utilities._

2. Configure Twitter credentials using twitter.txt

setupTwitter()

The implementation of setupTwitter is as below. It configures Twitter service credentials using twitter.txt in the main workspace directory. Here, the twitter credentials are set by calling System.setProperty(Twitter4J’s default OAuth authorization)

def setupTwitter() = {

import scala.io.Source for (line <- Source.fromFile(“../twitter.txt”).getLines) {

val fields = line.split(“ “)

if (fields.length == 2) {

System.setProperty(“twitter4j.oauth.” + fields(0), fields(1))

}

} }

3. StreamingContext is the main entry point for all streaming functionality. Set up a Spark streaming context, ssc, named “DonateBlood” that runs locally using all CPU cores and one-second batches of data. It receives stream data as input and produces batches of data.

val ssc = new StreamingContext(“local[*]”, “DonateBlood”, Seconds(1))

4. set the log level to ERROR to eliminate the noisy log Spark generates

setupLogging() def setupLogging() = {

import org.apache.log4j.{Level, Logger}

val rootLogger = Logger.getRootLogger()

rootLogger.setLevel(Level.ERROR) }

5. Create a DStream from Twitter using the Spark streaming context and the filters. DStream is a continuous data stream, which the user can analyze.

val tweets = TwitterUtils.createStream(ssc, None, filters)

Note:

a) We are passing ‘None’ as Twitter4J authentication because the twitter credentials are already set by calling setupTwitter()

b) We have created an immutable list of the filter keywords to pass as arguments while creating twitter stream

val filters = List(“blood”,”plasma”, “#blood”, “#bloodrequired”,”#BloodMatters”,”#BloodDrive”,”#DonateBlood”,”#Blood”,”#NeedBlood”)

Analyzing Tweets

1. Now extract the id, date, user, text, location, retweet, hashtags from each tweet using map(). Note that for extracting hashtags, we create a list of words by splitting the text of the status using ‘ ‘(space) and then filter only the words that start with #

val statuses = tweets.map{status =>

val id = status.getUser.getId

val user = status.getUser.getName

val date = status.getCreatedAt.toString

val text = status.getText

val location = status.getUser.getLocation

val retweet = status.isRetweet() val hashtags = status.getText.split(“ “).toList.filter(word => word.startsWith(“#”))

(id, date, user,text,location, retweet, hashtags) }

2. RDD transformation:

a) statuses.foreachRDD((rdd, time): Foreachrdd is used to perform an arbitrary action on each RDD in a DStream and also to save Dstream to external storage, in this case, a parquet file and a temporary table.

b) SparkSession.builder(): To use DataFrames and SQL operations on streaming data, we have to create a SparkSession using the SparkContext that the StreamingContext is using.

c) Convert RDD[String] to DataFrame using rdd.toDF() and filter out only the text that contains the words like urgent, need, emergency and required. This is for filtering only those messages where there is an urgent need for blood or plasma. This acts as an effective filter to filter general blood donation related messages like “I donated blood today”, “organizing a blood donation drive” etc.

d) Drop the duplicates using dropDuplicates(). This filters out the duplicates which occurs due to retweeting the same status.

e) rdd.count() > 0 and bloodDonateTweets.count() > 0: Filters out the empty batches

f) Finally, we save the resultant tweets as a temporary table to provide SQL Interoperability and also as a parquet file.

statuses.foreachRDD((rdd, time) =>

{

val spark = SparkSession.builder().appName(“MyProject”).getOrCreate() import spark.implicits._ if (rdd.count() > 0){ val requestsDataFrame = rdd.toDF(“id”,”date”,”user”,”text”,”location”,”retweet”,”hashtags”) val bloodDonateTweets = requestsDataFrame.filter(col(“text”).contains(“urgent”) || col(“text”).contains(“need”) || col(“text”).contains(“emergency”) || col(“text”).contains(“required”)).dropDuplicates(“text”) if (bloodDonateTweets.count() > 0){ bloodDonateTweets.createOrReplaceTempView(“BloodDonationTable”)

val solution = spark.sqlContext.sql(“select * from BloodDonationTable order by date”)

solution.show(false)

solution.coalesce(1).write.mode(SaveMode.Append).parquet(path) } } })

3. Kick it all off!

a) ssc.checkpoint(“C:/checkpoint/”) — We have enabled checkpoint by setting a directory in a fault-tolerant, reliable file system (e.g., HDFS, S3, etc.) to which the checkpoint information will be saved.

b) ssc.start() — Till this point the actual execution of the code does not start. After start() the JobScheduler starts, this in turn starts JobGenerator which creates the jobs.

c) ssc.awaitTermination()- it just waits for the termination signal from user. When it receives signal from user (i.e CTRL+C or SIGTERM) then it streaming context will be stopped.

ssc.checkpoint(“C:/checkpoint/”)

ssc.start()

ssc.awaitTermination()

Output:

I ran this code in Scala eclipse IDE. After a few minutes, I terminated the process, here are the results.

As you can see below, the ID, date, user, text, location, retweet, and the hashtags in the tweet are displayed.