DataSet from VADER for Sentiment analysis to fine tune Neural Language Model and CI Pipeline
Tags: Sentiment Analysis, NLP, Spark Streaming, Continuous Delivery, Python

Need to choose the live data feed for selecting the dataset (in this case using twitter data feed by subscribing to the data feed). Using VADER sentiment Analyser we can extract the positive / negative scores. As some of the outcome from VADER analyser would have high neutral score, the filtering process selects the data using compound score with the criteria (compound_score > 0.05 or< 0.05) as a starting point. Finally the processed data is sent to Sink (file system / datbase)

Sample code to extract the tweet data using tweepy library:

consumer_key=""

consumer_secret=""

access_key = ""

access_secret = "" #India geo area co-ordinates

GEOBOX_INDIA = [68.116667, 8.066667, 97.416667, 37.100000] filter_text = "india"

file = "./tweets_data.csv" #OAuth setup for tweepy auth = tweepy.OAuthHandler(consumer_key, consumer_secret)

auth.set_access_token(access_key, access_secret)

api = tweepy.API(auth) #listener for consuming twitter stream.

def twitter_listener(file_name,

filter_track,

follow=None,

locations=None,

languages=None,

time_limit=20):

class TwitterStreamListener(tweepy.StreamListener):

def __init__(self, time_limit):

self.start_time = time.time()

self.limit = time_limit

super(TwitterStreamListener, self).__init__()

def on_status(self, status):

if (time.time() - self.start_time) < self.limit:

# Writing status data

with open(file_name, 'a') as f:

writer = csv.writer(f)

writer.writerow([

status.author.screen_name,status.created_at,

status.text

])

else:

return False

def on_error(self, status_code):

if status_code == 420:

return False

else:

return True # Don't kill the stream

def on_timeout(self):

return True



with open(file_name, 'w') as f:

writer = csv.writer(f)

writer.writerow(['author', 'date', 'text'])

streamingAPI = tweepy.streaming.Stream(

auth, TwitterStreamListener(time_limit=time_limit))

streamingAPI.filter(

track=filter_track,

follow=follow,

locations=locations,

languages=languages,

)

f.close() filters=[filter_text]



twitter_listener(filter_track=filters, file_name=file, locations=GEOBOX_INDIA, languages=['en'], time_limit=60)

Sample code to process the data with Vader sentiment analyser

import glob

import pandas as pd

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer def sentiment_analyzer_scores(sentence):

score = analyser.polarity_scores(sentence)

return score def updateScores(sentence):

scores = sentiment_analyzer_scores(str(clean_tweets(sentence)))

return pd.Series((scores[‘pos’], scores[‘neg’],

scores[‘compound’],scores[‘neu’] ))df_tweet = #Read tweet data from file system and update sentiment analysis score and create label out of that. pd.concat([pd.read_csv(f, encoding=’latin1', sep=’\t’) for f in glob.glob(‘<path>/tweet-*.csv’)]) df_tweet[[‘pos’, ‘neg’, ‘compound’, ‘neu’]] = df_tweet.apply(lambda row: updateScores(row[‘text’]), axis=1) df_tweet_pos_neg[‘label’] = np.where(df_tweet_pos_neg[‘compound’] >= 0, 1, 0)

Next step is to Fine tune the language Model based on the extracted data set for sentiment analysis.

The pre labeled data set is processed and data is cleaned up and tokenized for fine tuning the language model. Then few Neural net layers to classify for sentiments be added along with the pre available language model. Once the language model is fine tuned for the expected accuracy, the language model with the weights are stored.

And the final stage is to use the fine tuned model to predict for the new live feed from twitter stream.

Similar to the first stage, twitter data is consumed and pushed to Kafka input topic. A python module with the fine tuned LM listens to the kafka input topic and process the tweet and predicts the sentiments Once the sentiment is predicted the results are published to Kafka output topic. Apache Spark Streaming process aggregates the sentiment results on a timely basis.

Sample code to aggregate the result is below using Apache Spark,

import org.apache.spark.sql.{Row, SparkSession}

import org.apache.spark.sql.types._

import org.apache.spark.sql.functions._

import org.apache.spark.sql.streaming.{OutputMode, Trigger}



import scala.concurrent.duration._

import net.liftweb.json._



object StructuredStreaming extends App {



val spark = SparkSession

.builder

.appName(“Spark-Kafka-Integration”)

.master(“local”)

.getOrCreate()



import spark.implicits._

// Message schema

val messageSchema = StructType(List(StructField(“id”, LongType), StructField(“text”, StringType),

StructField(“createdAt”, LongType), StructField(“retweet”, BooleanType),

StructField(“result”, IntegerType)))

val df = spark

.readStream

.format(“kafka”)

.option(“kafka.bootstrap.servers”, “localhost:9092”)

.option(“startingoffsets”, “earliest”)

.option(“subscribe”, “tweet-sentiment”)

.load()



implicit val formats = DefaultFormats



//Transform the json to dataframe.

val tweetMessages = df.select(from_json($”value”.cast(“string”),

messageSchema).alias(“tweet”))

.select($”tweet



//Group the sentiments by positive / negative and by time window.

val result = tweetMessages.withWatermark(“timestamp”, “1 minutes”).groupBy($”result”,

window($”timestamp”, “2 minutes”, “2 minutes”)).count() // Consume the generated sentiments of Language model.df = spark.readStream.format(“kafka”).option(“kafka.bootstrap.servers”, “localhost:9092”).option(“startingoffsets”, “earliest”).option(“subscribe”, “tweet-sentiment”).load()formats = DefaultFormats//Transform the json to dataframe.tweetMessages = df.select(from_json($”value”.cast(“string”),messageSchema).alias(“tweet”)).select($”tweet .id ”, $”tweet.result”,($”tweet.createdAt”/1000).cast(TimestampType).alias(“timestamp”))//Group the sentiments by positive / negative and by time window.result = tweetMessages.withWatermark(“timestamp”, “1 minutes”).groupBy($”result”,window($”timestamp”, “2 minutes”, “2 minutes”)).count() //Data sink to sent the aggregated results to another kafka topic.

val query = result

.selectExpr(“CAST(window AS STRING) AS key”, “to_json(struct(*)) AS value”)

.writeStream

.outputMode(“update”)

.format(“kafka”)

.option(“kafka.bootstrap.servers”, “localhost:9092”)

.option(“checkpointLocation”, “<checkpoint-path>”)

.option(“topic”, “sentiment-by-2-min”)

.trigger(Trigger.ProcessingTime(2.minutes))

.start()

.awaitTermination()

As each of the step is independent, these individual modules could be deployed independently using Jenkins /docker container which is for another post.