Twitter JSON data processing. Cleaning and polishing a dataframe of... | by Héctor Ramírez, Ph.D. | Towards Data Science

Twitter allows collecting tweets using tweepy, a Python library for accesing the Twitter API. Here I don’t intend to give a tutorial on how to collect tweets, as there are already several good reviews (take a look at the references below), but rather to give you a full example of how to process the tweet objects in order to build a clean dataframe on which we can perform social media analyses.
TL;TR: Along the way, we will flatten the Twitter JSON, select the text objects among the several options (main tweet, re-tweet, quote, etc.), clean them (remove non-alphabetic characters), translate non-English tweets, compute the sentiment of the text, and associate a location given a user-defined location or the automatic geolocalization.
Libraries to use: pandas, country converter, GeoPy, spaCy, googletrans, NLTK.
Each tweet object comes in JSON format, a mix of ‘root-level’ attributes, and child objects (which are represented with the {} notation). The Twitter developer page gives the following example:
{ "created_at": "Wed Oct 10 20:19:24 +0000 2018", "id": 1050118621198921728, "id_str": "1050118621198921728", "text": "To make room for more expression, we will now count all emojis as equal—including those with gender‍‍‍ ‍‍and skin t... https://t.co/MkGjXf9aXm", "user": {},   "entities": {}}
This is of course a small sample out of the huge dictionary composing each tweet. Another popular example is this Twitter Status Object map.
For most kinds of analyses, we will surely need attributes as the tweet text, the user screen-name or the tweet place. Unfortunately, as you can see, these attributes don’t come in a clean format, instead they are spread across the JSON levels — e.g., the tweet location coordinates are located in
tweet_object['place']['bounding_box']['coordinates']
It is due to this fact that the collected tweets need a large process of cleaning and transforming, which is the purpose of this post.
I recently carried out a language localization project where I needed to do a social media analysis on Twitter. For this, I collected 52830 tweets over the course of several days containing the following keywords: ‘#FIFA20’, ‘#FIFA21’, ‘FIFA20’, ‘FIFA21’, ‘FIFA 20’, ‘FIFA 21’ and ‘#EASPORTSFIFA’. Then, in order to do a proper analysis on them, I had to previously clean each tweet object so I could draw meaningful conclusion.
Due to the nature of that project, I was mainly interested in data regarding the location of the tweet (country and coordinates), the sentiment of the English version of the text, and the language the text was tweeted in. And it was the goal of the processing steps to polish and find this attributes. You can find the details of the project in the following repository:
github.com
Let’s use this dataset to exemplify the steps of tweets processing!
As we saw, there are multiple fields in the Twitter JSON which contains textual data. In a typical tweet, there’s the tweet text, the user description, and the user location. In a tweet longer than 140 characters, there’s also the extended tweet child JSON. And in a quoted tweet, there’s the original tweet text and the commentary with the quoted tweet.
To analyze tweets at scale, we will want to flatten the tweet JSON into a single level. This will allow us to store the tweets in a DataFrame format. To do this, we will define the function flatten_tweets() which will take several fields regarding text and location (this one stored in place). Take a look:
Now, you may want to study all the text fields (main, re-tweet or quote), however, here I will keep just one text field for simplicity. For this, we now define a function select_text(tweets) that selects the main text whether the tweet is a principal tweet or a re-tweet, and we decide to drop the quoted text as it usually is repetitive and may not be informative.
We now build the data frame. Notice that we choose the main columns (fields) relevant for a social media analysis. This includes the tweet language, lang, and the user-location, which is set manually by the user. We also keep the country, country_code and coordinates fields from place. These fields appear when the tweet is geo-tagged and it is usually contained in less than the 10% of the total of tweets. The following code block builds the dataframe:
import pandas as pd# flatten tweetstweets = flatten_tweets(tweets_data)# select texttweets = select_text(tweets)columns = ['text', 'lang', 'user-location', 'place-country',            'place-country_code', 'location-coordinates',            'user-screen_name']# Create a DataFrame from `tweets`df_tweets = pd.DataFrame(tweets, columns=columns)# replaces NaNs by Nonesdf_tweets.where(pd.notnull(df_tweets), None, inplace=True)
The head of the dataframe would look like:
Notice how only almost half of the tweets include a manually-set user-location field and not even the 1% are geo-tagged, i.e., they don’t provide a place field. This highlights the importance of collecting as many tweets as you can!
In the following, we are interested in cleaning and polishing each of the dataframe columns.
In this part of this process we will replace the languages codes in lang by the language standard name. As mentioned in the documentation:
When present, [lang] indicates a BCP 47 language identifier corresponding to the machine-detected language of the Tweet text, or und if no language could be detected.
We perform this step using the auxiliary languages.json file in this repository. This file maps the language code to the language standard names. The following code will do the trick:
with open('languages.json', 'r', encoding='utf-8') as json_file:    languages_dict = json.load(json_file)names = []for idx, row in df_tweets.iterrows():    lang = row['lang']    if lang == 'und':        names.append(None)    elif lang == 'in':        name = languages_dict['id']['name']        names.append(name)    elif lang == 'iw':        name = languages_dict['he']['name']        names.append(name)    else:        name = languages_dict[lang]['name']        names.append(name)df_tweets['language'] = namesdf_tweets.drop(['lang'], axis=1, inplace=True)
Now we move to process the locations. We will first treat theplace fields and then theuser-location field.
The data in the place object is ––obiously–– more reliable than the user-location. Therefore, although it constitutes the 0.91% of our tweets, we will take care of it. First, the country code in place-country_code comes in ISO 2 form, for which we will translate it to ISO 3 form with country converter. Then, we will perform the same to change place-country names to the standard, short names. This is advantageous because, for instance, Plotly maps use ISO 3 codes to localize countries.
Here we take the manually-set user-locations and translate them to country names and codes –– this involves some trusting on the user. We do this using the GeoPy library to identify a location (which could be an address) and assign a country to it. Also, again, we use country_converter to find the country codes in ISO 3 form.
A word of caution: GeoPy connects to an API and, unfortunately, it takes almost a second for each call. This makes the process of computing ~ 50 K tweets rather slow.
Note: tqdm is a python library, with an excellent implementation to pandas, that outputs a progress bar while the code is running. It’ll make your life easier!
Finally, we reduce the place-country and user-country columns to one by keeping the former when it exists, otherwise we keep the latter. We do the same for the codes columns:
countries, codes = [], []for idx, row in df_tweets.iterrows():    if row['place-country_code'] is None:        country = row['user-country']        code = row['user-country_code']        countries.append(country)        codes.append(code)    else :        countries.append(row['place-country'])        codes.append(row['place-country_code'])df_tweets['location'] = countriesdf_tweets['location_code'] = codes# drop old columnsdf_tweets.drop(columns=['place-country', 'place-country_code',                  'user-country', 'user-country_code'], inplace=True)
At this point, our dataset looks like this:
It is now time to process the tweets’ text. This will involve removing non-alphabetic characters and translate non-English tweets. We will however retain both options and actually use the texts with emojis and other characters for analysis given that our sentiment analyzer can handle them very well.
To remove non-alphabetic characters, we use spaCy as it is quite straightforward and we do not need to specify the regular expression. Keep in mind that the following block removes emojis and words with apostrophes like “I’m”, “y’all”, “don’t”, etc.
import spacynlp = spacy.load('en_core_web_sm')def cleaner(string):        # Generate list of tokens    doc = nlp(string)    lemmas = [token.lemma_ for token in doc]    # Remove tokens that are not alphabetic     a_lemmas = [lemma for lemma in lemmas if lemma.isalpha()                  or lemma == '-PRON-']     # Print string after text cleaning    return ' '.join(a_lemmas)df_tweets['text_cleaned'] = \                   df_tweets['text'].progress_apply(cleaner)
To translate the non-English tweets, we use googletrans which, as GeoPy, connects to its API, however it is much faster.
Another word of caution: It exists a poorly documented error discussed, e.g., here: https://stackoverflow.com/questions/49497391/googletrans-api-error-expecting-value-line-1-column-1-char-0, which makes disconnects you and blocks your IP. To bypass this error, I use np.array_split() to divide the dataframe into several chunks and process each of them at a time in a loop. By doing this, the error doesn’t occur, but I still save each chunks' translations to a csv so if in any iteration something went wrong, I can recompute just one chunk. I also instantiate Translator() each time.
And we finally append the original, unprocessed English texts to text_english:
# replaces NaNs by Nonesdf_english.where(pd.notnull(df_english), None, inplace=True)# add original English tweets to text_english by replacing Nonestexts = []for idx, row in df_english.iterrows():    if row['text_english'] is None:        text = row['text']        texts.append(text)    else :        texts.append(row['text_english'])df_english['text_english'] = texts
At this point, the dataframe looks like:
We finally compute the sentiment of each tweet. For this, we use NLTK’s SentimentIntensityAnalyzer object from the nltk.sentiment.vader library.
VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media. [Ref.]
This library is super simple to use, as you can see:
Notice that polarity_score outputs the probability that the text is negative, neutral or positive and a compound score. We then extract the latter and append the scores to the dataframe.
For the sake of presentation, we re-order the columns.
cols_order = ['text', 'language', 'location', 'location_code',               'location-coordinates', 'sentiment', 'text_english',               'text_cleaned', 'user-screen_name']df_final = df_sentiment[cols_order]
The final dataset should look like this:
To exemplify what could be done with this dataset, let’s build a visualization of the mean tweet sentiment score by country:
Notice that we used a country/languages dataframe which can be found in this repository. The code above outputs the following Plotly map:
This world map doesn’t look very positive 😕
The full code used in this post can be found in my repository:
github.com
I recently got my PhD in Physics, and I’m currently making my way into the data science world. Any comment and/or suggestion about this post is highly appreciated. Also, take a look at my other stories:
towardsdatascience.com
Finally, feel free to connect with me in LinkedIn:
www.linkedin.com
Datacamp — Analyzing Social Media Data in Python: