Finding the most popular hashtags on Twitter using Spark Streaming
By: Hritik Attri
Tags: Spark, Big Data, API, Python, Twitter

Photo by Markus Spiske on Unsplash

Spark Streaming is a component of Apache Spark. It is used to stream data in real-time from different data sources. In this section, we will use Spark Streaming to extract popular hashtags from tweets. The complete code implementation in Scala can be found at my GitHub page here: https://github.com/HritikAttri/big_data

Working of Spark Streaming can be quickly explained as follows:

Streaming Context: It receives stream data as input and produces batches of data. Discretized Stream(DStream): It is a continuous data stream, which the user can analyze. It is represented by a continuous set of RDDs and each RDD represents stream data from a specific time interval. The receiver receives streaming data and converts it into Input DStreams which can be used for processing. Certain transformations can be applied to DStream objects. Output DStreams are used to export data to external databases for storing it. Caching: If the streaming data is to be computed multiple times, it is better to persist it using persist(). It will load this data in memory and by default, data is persisted two times in memory for backup in case of failure. Checkpoints: We create checkpoints at certain intervals to rollback to that point in case of failure in the future.

Let’s begin by importing the libraries:

package com.spark.examples import scala.io.Source

import org.apache.log4j.{Level, Logger}

import org.apache.spark._

import org.apache.spark.SparkContext._

import org.apache.spark.streaming._

import org.apache.spark.streaming.twitter._

import org.apache.spark.streaming.StreamingContext._

Set the log level to print errors only.

def setupLogging() = {

val rootLogger = Logger.getRootLogger()

rootLogger.setLevel(Level.ERROR)

}

We need to set up the twitter configuration file. For this, one needs to have a Twitter Developer Account. Go to https://developer.twitter.com/en/apps . Fill the details, create an app, go to details, then keys and tokens and click on the regenerate button. We need the API key, API secret key, access token, access secret token to connect to Twitter and stream data. Find the twitter text file used here on my Github page. Again, the link is given at the beginning of this post.

def setupTwitter() = {

for(line <- Source.fromFile("C://twitter.txt").getLines) {

val fields = line.split(" ")

if(fields.length == 2) {

System.setProperty("twitter4j.oauth." + fields(0), fields(1))

}

}

}

Creating a StreamingContext which will stream batches of data every one second.

val ssc = new StreamingContext("local[*]", "PopularHashtags", Seconds(1))

Create a DStream object for Twitter streaming.

val tweets = TwitterUtils.createStream(ssc, None)

Extract hashtags from each tweet.

val text = tweets.map(x => x.getText())

val words = text.flatMap(x => x.split(" "))

val hashtags = words.filter(x => x.startsWith("#"))

Now, we will extract hashtags in real-time for 5 minutes, and in a window of 1 second each, we update the count of the hashtags.

val hashtags_values = hashtags.map(x => (x, 1)) val hashtags_count = hashtags_values.reduceByKeyAndWindow((x, y) => x + y, (x, y) => x - y, Seconds(300), Seconds(1))

Sort the results in descending order.

val results = hashtags_count.transform(x => x.sortBy(x => x._2, false))

Set up a checkpoint.

ssc.checkpoint("C://checkpoint") ssc.start() ssc.awaitTermination()

I ran this code in Scala eclipse IDE. After a few minutes, I terminated the process, here are the results.

As you can see, #BTS was the most popular hashtag during the code execution with a count of 86.