Build Dynamic Visualizations and Dashboards with Kibana for Data Analysis using Twitter Feeds
By: Andrew Eng
Tags: Osint, Data Analysis, Elasticsearch, Data Science, Visualization Tool

Build Dynamic Visualizations and Dashboards with Kibana for Data Analysis using Twitter Feeds

OSINT with Python and ELKstack [Part 2a]

Image by Author: Generated using https://www.wordclouds.com/

Get Synched

< Previous Story |[x] Current

In part 1 we:

Built our infrastructure using a containerized version of ELKSTACK (Elasticsearch, Logstash, Kibana) Explored and built a python script that reaches out to Twitter using APIs Transferred the collection data into Elasticsearch for analyzing

In part 2, we are going to explore the data that we ingested into Elasticsearch using Kibana.

While writing this part, I realized that this is going to be a huge topic. While going through EDA, I found so many useful tools and techniques that could be used to speed up the process. I’m going to break part 2 into multiple sub-sections. This sub-section is going to be focused on the initial EDA and building out simple visualizations with Kibana. The next sections will focus on cleaning up the data using machine learning and tap into other APIs that will help us answer some questions. There may be additional quesitons that we come up with through the process.

Since part 1, I expanded the search criteria to include all of the metadata from each tweet. That way, I can go back to it and answer possible new questions around the data I’m exploring. I need to clean it up a bit more. Note I need to create TRY exceptions around KeyError codes (some fields do not have data, but I still want to collect that field).

Since the code was getting larger than I expected. I took out some uneeded things. I simplified the feed to elastic function:

while count < len(feed):

doc = {

'@timestamp': dt.now(),

'created_at': str(feed[count]['created_at']),

'twitter_id' : int(feed[count]['id']),

'id_str' : int(feed[count]['id_str']),

'full_text' : str(feed[count]['full_text']),

'truncated' : str(feed[count]['truncated']),

'display_text_range' : str(feed[count]['display_text_range']),

'entities' : str(feed[count]['entities']), # Already split the dictionary, no longer needed

'metadata' : str(feed[count]['metadata']),

'source' : str(feed[count]['source']),

'in_reply_to_status_id' : str(feed[count]['in_reply_to_status_id']),

'in_reply_to_status_id_str' : str(feed[count]['in_reply_to_status_id_str']),

'in_reply_to_user_id' : str(feed[count]['in_reply_to_user_id']),

'in_reply_to_user_id_str' : str(feed[count]['in_reply_to_user_id_str']),

'in_reply_to_screen_name' : str(feed[count]['in_reply_to_screen_name']),

'user' : str(feed[count]['user']), # Already split the dictionary, no longer needed

'geo' : str(feed[count]['geo']),

... [truncated]

This clean up was pretty fun. I was doing it manually, but it was taking forever. [VERY tedius] I had to open the dictionary, extract the keys, format the text so it creates variables and then those variables are called. So instead… I turned to python. I had to think of ways I can keep iterating through dictionaries and sub-dictionaries to extract the items for each key. I want a simple dictionary that I can stat on. Note Clean up the collection function:

I moved some code blocks around and wrapped it in a function that I can use to call. I did this because it provides better flexibility when try/exceptions are being executed.

This code block imports modules, sets server and API configurations, and objects

# Import tweepy as tw

import tweepy as tw

import sys

from datetime import datetime as dt

from elasticsearch import Elasticsearch # Initializing objects

twitter_cred = dict()

api = ''

es = '' def setConfig(server):

# Import keys from a saved file instead of inputting it directly into the script.

# Strip whitespaces and split on = as I only want the key values

# Server argument is the elasticsearch node



key_location = 'twitter.keys'

apikeys = [] global api

global es with open(key_location) as keys:

for i in keys:

apikeys.append(i.split("=")[1].strip(" ").strip("

"))

keys.close() # Initialize dictionary

#twitter_cred = dict() # Enter API keys

twitter_cred["CONSUMER_KEY"] = apikeys[0]

twitter_cred["CONSUMER_SECRET"] = apikeys[1] # Access Tokens

twitter_cred["ACCESS_KEY"] = apikeys[2]

twitter_cred["ACCESS_SECRET"] = apikeys[3] # Set authentication object

auth = tw.OAuthHandler(twitter_cred["CONSUMER_KEY"], twitter_cred["CONSUMER_SECRET"])

auth.set_access_token(twitter_cred["ACCESS_KEY"], twitter_cred["ACCESS_SECRET"]) # Create api object with authentication

api = tw.API(auth, wait_on_rate_limit=True) # Set Elasticsearch Server

es = Elasticsearch(server, port=9200) # Execute function with the elasticsearch ip address

setConfig('127.0.0.1')

Pull a tweet so we can play around with the record

feed = {} for tweet in tw.Cursor(api.search, q='palantir OR pltr', tweet_mode='extended').items(1):

feed.update(tweet._json)

Iterate the dictionary and sub dictionary to generate the line(s) of code I need

# What keys are used

parentList = []

subList = [] # Add dictionaries to sub process list

def subProcess(dictionary):

subList.append(dictionary)



for item in feed.keys():

parentList.append(item) for i in parentList:

if type(feed[i]) is not dict:

print(f"{i} = feed['{i}']")



else:

subProcess(i) ----- output ----- created_at = feed['created_at']

id = feed['id']

id_str = feed['id_str']

full_text = feed['full_text']

truncated = feed['truncated']

display_text_range = feed['display_text_range']

source = feed['source']

in_reply_to_status_id = feed['in_reply_to_status_id']

in_reply_to_status_id_str = feed['in_reply_to_status_id_str']

in_reply_to_user_id = feed['in_reply_to_user_id']

in_reply_to_user_id_str = feed['in_reply_to_user_id_str']

in_reply_to_screen_name = feed['in_reply_to_screen_name']

geo = feed['geo']

coordinates = feed['coordinates']

place = feed['place']

contributors = feed['contributors']

is_quote_status = feed['is_quote_status']

retweet_count = feed['retweet_count']

favorite_count = feed['favorite_count']

favorited = feed['favorited']

retweeted = feed['retweeted']

possibly_sensitive = feed['possibly_sensitive']

lang = feed['lang']

Exploratory Data Analysis with Kibana

Getting Started

Exploratory Data Analysis (EDA) is the process for performing initial investigations on data. The goal is to discover patterns, outliers, and develop hypothesises. Visualizations can be created to help streamline this process.

“Kibana is a free and open source user interface that lets you visualize your Elasticsearch data and navigate the Elastic Stack. Do anything from tracking query load to understanding the way requests flow through your apps.”

Image Sourced from elastic.co

I use Kibana exclusively for EDA and creating visualizations in order to try and understand data. As I progress, I imagine I will be using python pandas and matplotlib more to quickly sift through data. For now, I’ll stick with Kibana.

Access the notebook

# Code snippet will not be included in the blog to reduce the read time. This will be a supplement in my github repository import tweepy as tw

import sys

from datetime import datetime as dt

from elasticsearch import Elasticsearch # Initializing objects

twitter_cred = dict()

api = ''

es = '' def acqData(search, acq):

feed = []



...[Truncated]

Get the source code at: index_name = idx + dt.today().strftime('%Y-%m-%d')feed = []...[Truncated]Get the source code at: github

For the purpose of this notebook, the below cell is a modified version that allows me to just execute the functions.

try:

idx = ‘default-’

tweet_count = 100

search = ‘palantir OR PLTR’ setConfig(‘127.0.0.1’)

acqData(str(search), int(tweet_count))



except:

pass ---- output ---- : :Acquiring Data::

: :Transferring to Elasticsearch Search::

Setup Kibana Index

Great! We got data into Elasticsearch. Now, let’s create a Kibana index of it:

Image by Author

1. Browse to http://127.0.0.1:5601 > Click on “Explore on my own”

Image by Author

2. Click “Stack management”

Image by Author

3. Create Index

Image by Author

4. Match Index Patterns

Image by Author

5. Type “default*”

Image by Author

6. Sort by “@Timestamp”

Image by Author

fin

Image by Author

Click Discover

Image by Author

Select “This Week”

The initial questions we are trying to answer are:

What are the most retweets of the specific search term? (Tracking topic popularity)

How many unique users are tweeting about a given search term? (Is it trending?)

What other tweets are they posting about the search term? (Are they a troll / bot?)

How many likes does the person have on the search term? (Trending / Sentiments)?

The following fields seems interesting when Scanning through the “Available Fields”:

full_text

lang

retweet_count

user_created_at

user_screen_name

user_followers_count

user_friends_count

user_statuses_count

user_created_at

I added lang field in so I can later figure how what is the sentiments per “language” ratio. For now, I’m filtering out non-English.

Image by Author: Filter English only

Filtering english-only reduces my sample from 400 to 262.

Scanning through the full_text, I see there are a bunch of retweets that I want to aggregate into 1. The stats show that the top 5 values are retweets and accounted for X percentage of the 262 tweets. We see that the top retweet accounted for 19.1%. Let’s explore that.

Image by Author

I got super excited as I thought it was that easy and found all the bots. However, a closer look. retweet_count is NOT the amount of times the user_screen_name retweeted. It’s how many times the full_text was retweeted.