Twitter JSON data processing
By: Héctor Ramírez
Tags: Social Media Analysis, Sentiment Analysis, NLP, Twitter, Data Science

Each tweet object comes in JSON format, a mix of ‘root-level’ attributes, and child objects (which are represented with the {} notation). The Twitter developer page gives the following example:

{

"created_at": "Wed Oct 10 20:19:24 +0000 2018",

"id": 1050118621198921728,

"id_str": "1050118621198921728",

"text": "To make room for more expression, we will now count all emojis as equal—including those with gender‍‍‍ ‍‍and skin t… https://t.co/MkGjXf9aXm",

"user": {},

"entities": {}

}

This is of course a small sample out of the huge dictionary composing each tweet. Another popular example is this Twitter Status Object map.

For most kinds of analyses, we will surely need attributes as the tweet text, the user screen-name or the tweet place. Unfortunately, as you can see, these attributes don’t come in a clean format, instead they are spread across the JSON levels — e.g., the tweet location coordinates are located in

tweet_object['place']['bounding_box']['coordinates']

It is due to this fact that the collected tweets need a large process of cleaning and transforming, which is the purpose of this post.

The Twitter data

I recently carried out a language localization project where I needed to do a social media analysis on Twitter. For this, I collected 52830 tweets over the course of several days containing the following keywords: ‘#FIFA20’, ‘#FIFA21’, ‘FIFA20’, ‘FIFA21’, ‘FIFA 20’, ‘FIFA 21’ and ‘#EASPORTSFIFA’. Then, in order to do a proper analysis on them, I had to previously clean each tweet object so I could draw meaningful conclusion.

Due to the nature of that project, I was mainly interested in data regarding the location of the tweet (country and coordinates), the sentiment of the English version of the text, and the language the text was tweeted in. And it was the goal of the processing steps to polish and find this attributes. You can find the details of the project in the following repository:

Let’s use this dataset to exemplify the steps of tweets processing!

Processing the JSON

As we saw, there are multiple fields in the Twitter JSON which contains textual data. In a typical tweet, there’s the tweet text, the user description, and the user location. In a tweet longer than 140 characters, there’s also the extended tweet child JSON. And in a quoted tweet, there’s the original tweet text and the commentary with the quoted tweet.

To analyze tweets at scale, we will want to flatten the tweet JSON into a single level. This will allow us to store the tweets in a DataFrame format. To do this, we will define the function flatten_tweets() which will take several fields regarding text and location (this one stored in place ). Take a look:

Now, you may want to study all the text fields (main, re-tweet or quote), however, here I will keep just one text field for simplicity. For this, we now define a function select_text(tweets) that selects the main text whether the tweet is a principal tweet or a re-tweet, and we decide to drop the quoted text as it usually is repetitive and may not be informative.

We now build the data frame. Notice that we choose the main columns (fields) relevant for a social media analysis. This includes the tweet language, lang , and the user-location , which is set manually by the user. We also keep the country , country_code and coordinates fields from place . These fields appear when the tweet is geo-tagged and it is usually contained in less than the 10% of the total of tweets. The following code block builds the dataframe:

import pandas as pd



# flatten tweets

tweets = flatten_tweets(tweets_data)



# select text

tweets = select_text(tweets)

columns = ['text', 'lang', 'user-location', 'place-country',

'place-country_code', 'location-coordinates',

'user-screen_name']



# Create a DataFrame from `tweets`

df_tweets = pd.DataFrame(tweets, columns=columns) # replaces NaNs by Nones

df_tweets.where(pd.notnull(df_tweets), None, inplace=True)

The head of the dataframe would look like: